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Abstract 

Researchers and policymakers are increasingly interested in the issue of school 
accountability. Despite this, program standards for afterschool programs are not as fiilly 
developed as they are in other fields. This study bridges that gap and presents the results 
from a study that identifies benchmarks and indicators for high quality afterschool 
programs. This research employed a multi-method approach, including a synthesis of 
literature on afterschool programs, observations, and a survey data collection of 1 5 high- 
quality afterschool program sites. Results of the study suggest that most of the issues 
emphasized in the afterschool literature can be considered core components of a quality 
afterschool program. This finding was consistent across the three broad categories of 
program organization, program environment, and instructional features. This study also 
revealed that some issues emphasized in the afterschool literature should be considered 
extra components that can increase quality, but that are not necessary. As a result, this 
study argues for a checklist strategy in assessing programs in order to meet quality-based 
standards. With further testing, refinement, and validation from larger study samples, this 
checklist tool can help evaluate afterschool programs in order to not only obtain basic 
core standards, but also to assist in identifying and tackling weak and problematic areas. 

Introduction 

The enactment of the No Child Left Behind Act of 2001 (NCLB; 2002) has led to 
increased nationwide demands for school accountability. In particular, NCLB calls for 
school-based efforts to close the achievement gap and to ensure that all students, including 
those who are disadvantaged, gain academic proficiency. Under NCLB, schools must provide 
parents and the community with annual reports about their academic progress. Schools that 
lack progress may use afterschool programs as a supplemental service to help students learn 
more effectively. Although afterschool programs were initially created as safe havens for 
students, NCLB reinforces the important role that afterschool programs can have in 
increasing students’ academic proficiency and school engagement. 

At the same time, state legislation continues to promote the value of afterschool 
programs for the youth of California. In September 2005, Governor Schwarzenegger signed 
into law SB 854 (Ashburn) which increased both per student and administrative funding for 
21st Century Community Learning Centers (California AfterSchool Network, 2005). 
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Furthermore, under the mandates of Proposition 49, California increased its yearly budget for 
afterschool programs from $120 to $550 million dollars during the 2006-2007 fiscal year 
(California AfterSchool Network, 2007).' Legislation directed at increasing funding for 
afterschool programs is clearly an important priority, but the ability to simultaneously fund 
quality programs is an effort that requires immediate attention. 

As published in the Governor’s Guide to Extra Learning Opportunities (ELO; Wright, 
2005), there is a call for “an accountability system for improved ELO quality” (p. 4); it is an 
opportunity for governors to “provide incentives for programs to meet upon agreed 
benchmarks of ELO quality” (p. 21). Reason being, the standards (or quality benchmarks) for 
afterschool programs, upon which these incentives would be based, are not as fully 
developed as they are in other fields. Based on a recent synthesis of the research, it was 
suggested that policymakers should: 

1. Set standardized expectations for afterschool programs to run efficiently and 
effectively. 

2. Consistently evaluate and improve upon the structure and implementation of 
afterschool programs; that is, examine what works and what needs to be changed in 
terms of organization, environment, and instruction of students. 

This effort is complex. In order for policymakers to set realistic expectations for 
afterschool programs and use evaluations appropriately, they require information about tested 
indicators of quality and meaningful recommendations for those programs that need 
improvement. 

Establishing a template for quality afterschool programs and generalizing the impact is 
difficult for several reasons: (a) not all programs serve children with similar characteristics 
(i.e., race, socioeconomic status, age); (b) different programs have different goals and 
approaches; and (c) many differ on the desired program outcome (e.g., academic 
achievement, enrichment, or drug use prevention). It is for these reasons that a flexible model 
is needed that include planning strategies and implementation steps broad enough to 
encompass the variability of existing afterschool programs, yet specific enough to include 
key components of quality in afterschool programming. In addition, in order for research to 
effectively inform current policy and funding allocations, the need continues for efforts on 
improving the quality of programs that target minority and low-income students. 

Therefore, the primarily purpose of this study is to identify and establish quality 
benchmarks for afterschool programs in the areas of program environment, program 

’ As mandated by Proposition 49, funding for afterschool programs was increased once the California state 
budget reached a level making the release of funds feasible (California AfterSchool Network, 2007). 
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organization, and instructional features; and secondarily, to present a preliminary tool for the 
application of an indicator system. The research questions for this study were; 

• What are the basic core benchmarks for quality afterschool programs? 

• What are the key indicators that help define the core benchmarks in afterschool 
programs? 

• How can afterschool programs use a data-based system to identify their own 
strengths and weaknesses and strive for eontinuous improvement? 

Grounded in existing research, the study reported here was intended to inform 
policymakers about ways (shown as benchmarks and indicators) in which afterschool 
programs could benefit students by implementing strategies and components that promote 
program success and improvement. With aceess to a benchmark and indicator system, 
policymakers could make research-based decisions so that state funding could be applied 
toward programs that demonstrate promise and success. Furthermore, policymakers eould 
also use this information to develop and implement appropriate guidelines for policy 
involving afterschool programs. Additionally, based on the findings of this study, an 
evaluative system could be developed to assist afterschool programs in the self-monitoring of 
their progress and demonstrate their effectiveness in aiding their students. Managers of 
afterschool programs could use this system to fine-tune their objectives and goals, and 
demonstrate aecountability for policymakers, as well as promote positive outcomes for their 
students. 

The purpose of this study was to identify benchmarks and indicators that could set 
standardized expectations for afterschool programs and apply the identified benchmarks and 
indicators in a data-based system that afterschool practitioners could use to identify their own 
strengths and weaknesses, and assist them in striving for continuous improvement. 
A synthesis of literature was eonducted to extract common indicators and benchmarks 
mentioned in the literature. The prevalence of these indicators and benchmarks were further 
examined in three high quality afterschool programs. Employing a weighting strategy, the 
established benehmarks and indicators were ineluded in the design of a Quality Benchmark 
Rating System. The purpose of this system was to serve as a quick self-reference guide to 
practitioners for self-improvement, and a tool for policymakers so that they could use the 
system as an instrument for “quality at a glance” data-based decision making. 

A literature review on quality indieators for afterschool programs are provided in the 
following seetion. 
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A Review of the Literature 



Researchers have found afterschool programs heneficial to student outcomes in three 
critical ways. First, they provide children with supervision during afterschool hours — a time 
period where research has found rates for both victimization of juveniles and juvenile crimes 
peak. (U.S. Department of Education & U.S. Department of Justice, 2000; Snyder & 
Sickmund, 1995). Secondly, they provide experiences that may benefit students’ social skills 
and work habits (Fashola, 1998). Finally, afterschool programs may help improve academic 
achievement through tutoring and enrichment activities (Fashola, 1998). Quality afterschool 
programs can provide these basic benefits to students, as well as additional opportunities to 
acquire new skills and broaden their educational experiences. Recognizing the potential of 
afterschool programs to have a positive impact on the academic and social development of 
students, it is important to assess the critical factors necessary in providing a quality 
afterschool program. 

Research suggests effective afterschool programs provide students with safety, 
opportunities for positive social development, and academic enrichment (Ff.S. Department of 
Education & Fl.S. Department of Justice, 2000; Clark, 1988; Hetherington, Stanley-Ragan, & 
Anderson, 1989; Benard, 1991; Posner & Vandell, 1994; Miller, 1995; Snyder & Sickmund, 
1995). The researchers build from these basic elements for effective programming to provide 
a more comprehensive view of the factors necessary for increasing quality in afterschool 
programs. Following an in-depth review of literature on quality afterschool programs, the 
researchers identified three major areas that can determine effectiveness. These areas include 
(a) program organization, (b) program environment, and (c) instructional features. The 
following sections provide a description of these three areas, as described by the literature, 
and identify key benchmarks for assessing and improving program effectiveness. 

Program Organization 

Research on quality afterschool programs consistently identifies strong program 
organization as a crucial element in effective programs (Alexander, 1986; Beckett, Hawken, 
& Jacknowitz, 2001; Fashola, 1998; Huang, 2001; C. S. Mott Foundation Committee on 
After-School Research and Practice, 2005; McElvain & Cap Ian, 2001; Philadelphia Youth 
Network, 2003; Schwendiman & Fager, 1999). In 2005, the C. S. Mott Foundation 
Committee on After-School Research and Practice suggested a “theory of change” 
framework for afterschool programs that explicitly links program organization and 
participant outcomes to program effectiveness and quality. An in-depth review of literature 
conducted for this study (see Appendix A) indicated that seven specific elements of program 
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organization were consistently referenced in research. They included; (a) program 
management, (b) program administration, (c) staff support, (d) staff experience and training, 
(e) family and community involvement, (f) community partnerships, and (g) evaluation. 

Program management and program administration. Effective program management 
is necessary for quality-based afterschool programs. Huang (2001) specified that effective 
program organization should include a strong team of program staff who demonstrate 
leadership skills, a positive organizational climate, and inclusive decision-making. More 
specifically, it is important to have leadership articulate a shared mission statement and 
program vision that motivates staff, provides a positive organizational climate that validates 
staff commitment to these goals, as well as open the communication channels between 
afterschool, day school, parent, and community (American Youth Policy Forum, 2006; 
Wright, Deich, & Szekely, 2006). Strong program management also provides adequate 
compensation for staff, thus decreasing the likelihood of high turnover rates (Beckett et al., 
2001; de Kanter, 2001; C. S. Mott Foundation, 2005). Moreover, a strong leadership team 
and committed staff must also plan for program sustainability and growth through effective 
administration (ERIC Development Team, 1998), including systematic organization of 
student records, program attendance, resource needs, program budget, a future financial plan, 
and marketing (St. Clair, 2004). 

Staff support. A strong management team that is committed to achieving program 
goals should provide their staff with adequate support to help them perform their duties. At 
the basic level, staff must be provided with sufficient materials in order to conduct program 
activities (St. Clair, 2004). A positive working environment, such as clear expectations for 
staff performance, a job orientation prior to beginning work, time and space to express 
concerns, continuous feedback on their performance, a shared decision-making process, and 
opportunities for staff members to collaborate and express their individual talents are all 
strategies that will promote a sense of belonging, self-efficacy, and provide opportunities for 
staff members to make an impact on program quality (Beckett et al., 2001). 

Staff experience and training. In order to enhance staff efficacy, the staff must have 
the appropriate experience and training in working with afterschool students (Alexander, 
1986; Harvard Family Research Project, 2005; Huang, 2001; Fashola, 1998; de Kanter, 2001; 
ERIC Development Team, 1998; Schwartz, 1996). For example, each staff member should 
be competent in core academic areas for the respective age groups that they work with. 
Beyond academic competency, the staff should also be culturally competent, knowledgeable 
of the diverse cultures and social influences that can impact the lives of the students in the 
program (Huang, 2001; Schwartz, 1996). When the demographics of program staff reflect the 



5 



diversity of the community in which the program is located, these staff members can better 
serve as mentors and role models to the student participants (Vandell & Shumow, 1999; 
Huang, 2001). To ensure high quality instruction, staff members should be consistently 
provided with opportunities for professional development (Wright, 2005). To demonstrate 
academic effects, it is also important for students in the program to have sufficient access to 
qualified staff — in order to ensure that each student is given sufficient attention, according to 
her or his individual needs. Thus, having an adequate staff- to- student ratio is an important 
indicator of quality for afterschool programs (Yohalem, Pittman, & Wilson- Ahlstrom, 2004). 

Family and community involvement. Research on afterschool programs consistently 
associates family and community involvement with program quality (Owens & Vallercamp, 
2003; Tolman, Pittman, Yohalem, Thomases, & Trammel, 2002). Afterschool programs can 
promote family and community involvement by setting defined plans to involve parents, 
family members, and community volunteers. For example, programs might organize 
orientation sessions for incoming students and their families. At these sessions, families can 
be introduced to different involvement opportunities. Meanwhile, staff can regularly 
communicate with parents and families in order to provide a clear channel of communication 
that keeps parents informed of their children’s progress within the program (American Youth 
Policy Forum, 2006; Wright et ah, 2006). With open communication, families may also feel 
more comfortable engaging with staff about how the program can better support the needs of 
the student participants. When family involvement is acknowledged and encouraged, families 
and staff are able to work together to ensure high quality programming (Chung, 2000; 
Tolman et al., 2002). 

Community partnerships. Beyond students’ families, the local community is another 
valuable resource for afterschool programs. Research shows that high quality programs are 
consistently engaged with local community members, leaders, and organizations that can 
form important partnerships in program planning and funding (Birmingham, Pechman, 
Russell, & Mielke, 2005; Harvard Family Research Project, 2005; Owens & Vallercamp, 
2003; Wright, 2005). Through these partnerships, students can further develop knowledge of 
community resources, services, and histories. In turn, students may be encouraged to 
participate in community service projects that can reflect a sense of empowerment and pride 
in their respective communities. Programmatic efforts to form community partnerships could 
include inviting community members as guest speakers and recruiting local volunteers. 

Evaluation. As an instrument to inform continuous self-improvement, periodic 
evaluations are critical for the sustainability of afterschool programs (Huang, 2001). 
Furthermore, having evidence of program outcomes is essential for continued and or 



6 



increased funding and support (Seott-Little, Hamann, & Jurs, 2002; Wright et ah, 2006). 
Therefore, evaluations should be administered regularly to ensure continuous improvement 
and assess program effeetiveness (C. S. Mott Foundation, 2005). 

Thus, high quality afterschool programs should have a detailed plan for evaluation of 
program activities, staff performance, and student development (Seppanen, et ah, 1993). 
Students’ academie improvement and soeial skills development ean be especially important 
in documenting program outcomes. Overall satisfaction evaluations can also be assessed 
among staff, students, and families to ensure that expeetations and the needs of all program 
participants are being met (Fashola, 1998). Evaluation findings should be consistently 
reviewed and made readily available in order to examine program progress. 

Program Environment 

The program environment focuses on how the structure of the afterschool program 
creates an atmosphere conducive to positive academic achievement and self-esteem for 
youth; they are “attractive affective contexts” for youth development (Kahne et ah, 2001, 
p. 421). The four main elements of the program environment, which are consistently 
referenced by the researeh, inelude (a) safe environment, (b) student health and well-being, 
(c) well-equipped/suitable physical space, and (d) positive relationships. 

Safe environment and well-equipped/suitable physical space. First and foremost, the 
most important feature of the program environment is safety and seeurity within the indoor 
and outdoor space. It is well documented that program space should be safe, clean, and 
seeure for eultivating eonfidenee and self-esteem for students (Chung, 2000; North Carolina 
Center for Afterschool Programs, n.d.; National Institute on Out-of-School Time, 2002; 
Philadelphia Youth Network, 2003; New Jersey School-Age Care Coalition, 2002; St. Clair, 
2004; Wright et ah, 2006); no potential harm should be placed upon the health and physical/ 
emotional well-being of students (Safe and Sound, 1999). Adequate and comfortable space is 
needed for staff members to conduct a range of aetivities that promote both the mental and 
physical wellness of students. The indoor and outdoor space should also be used 
appropriately; catering to the aetivity being carried out (e.g., sports, creative arts, eating), so 
that the goals of the activities are sufficiently met. In addition, there should be ample storage 
spaee for equipment, materials, and personal possessions. Equipment should be able to be 
stored for easy student aecess and availability. The main aim is to make sure that students are 
in a safe, supervised environment that provides ample resources for mental and physical 
growth. The establishment of a physically and emotionally safe environment thus helps the 
development of positive relationships within the program environment. 
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Student health and well-being. Another facet of the program environment is the need 
to promote student wellness through health and nutrition education (de Kanter, 2001; North 
Carolina Center for Afterschool Programs, n.d.; National Institute on Out-of-School Time, 
2002; Philadelphia Youth Network, 2003; Wright, 2005). Nutritional time in afterschool 
programs offer students time to share meals and socialize with their peers while developing 
healthy snack habits that enhance students’ well-being (Chung, 2000). Furthermore, quality- 
based afterschool programs also provide environments that enhance the well-being of 
students by educating students and providing them with nutritious snacks adequate to portion 
size; and instructing the staff to minimize the health risks of students (e.g., having students 
wash their hands, having frequent restroom breaks). Exposure to health and wellness 
practices in the program environment allow students to be active and more fully engaged in 
nutrition and fitness related activities in their own lives (Wright, 2005). 

Positive relationships. The emotional climate of the program environment is 
characterized by warm, supportive relationships between the staff members and students, 
among the students themselves, and between staff members. These three types of 
relationships within the program setting signify positive, influential connections for the 
students (Beckett et ah, 2001; Huang, 2001; Birmingham et ah, 2005). First, the interaction 
between the staff members and students is vital for demonstrating affirmative adult-student 
relationships, aside from primary-based interactions within the home (Carnegie Council on 
Adolescent Development, 1994; Beckett et ah, 2001; New Jersey School-Age Care Coalition, 
2002; Harvard Family Research Project, 2004; Birmingham et ah, 2005; Bodily & Beckett, 
2005). Quality-based afterschool programs are structured to have written guidelines for staff- 
student relations so that the staff members are able to set appropriate guidelines and limits for 
students through positive behavior management strategies. 

Secondly, staff members should be expected to be emotionally invested in the lives of 
their students. Quality-based programs foster this relationship by enforcing a small staff-to- 
student ratio that provides a “family-like” atmosphere and contributes to positive social 
development for students (Carnegie Council on Adolescent Development, 1994; National 
Association of Elementary School Principals, 1999; Chung, 1997, 2000; Beckett et ah, 2001; 
Bodily & Beckett, 2005). Staff members are able to form more personable, one-on-one 
relationships with students through daily conversations and engagement (St. Clair, 2004). 
Consequently, this initiates a sense of community and belonging for the students because 
they are personally bonded to staff members (Wright et ah, 2006). 

Thirdly, positive peer relationships and friendships are a key ingredient in shaping 
students’ social-emotional development (Safe and Sound, 1999; Huang, 2001; Pechman & 



Marzke, 2003; Halpern, 2004; Harvard Family Research Project, 2004; Yohalem et ah, 2004; 
Yohalem, Wilson- Ahlstrom, & Yu, 2005). Students need to interact with each other, building 
strong “partnerships” based on trust and respect with their peers (Yohalem et ah, 2004). 
Healthy interaction with other students of various ages, and being involved in age appropriate 
activities helps students to demonstrate appropriate problem solving strategies, especially 
during times of conflict (Wright et ah, 2006). 

Finally, the adult relationships between staff members are also important in 
constructing an emotional climate within the program environment. Students observe 
positive adult interactions through effective communication and cooperation of the staff in 
working together to meet the needs of students and the program (Yohalem et ah, 2005). This 
relationship is an appropriate way in which the staff can model positive behavior for 
students. Staff members, for that reason, need to embrace assessment-based improvement 
plans as “relevant, contextual, and potentially helpful” (Weisberg & McLaughin, 2004, p. 4). 
Staff members must see the relevance of quality-based standards in shaping positive 
developmental outcomes for students. 

Thus, the program environment within high quality afterschool programs should offer a 
safe, healthy, and nurturing environment for all participants. This includes a physical and 
social environment that fosters resilient outcomes through the reinforcement of positive 
relationships, nutrition, and physical/academic activities (Huang, 2001; New Jersey School- 
Age Care Coalition, 2002; St. Clair, 2004; Harvard Family Research Project, 2004). 

Instructional Features 

Afterschool programs vary greatly in their emphasis: ranging from providing 
supervision or tutoring, to the promotion of specific learning and development. Increasingly, 
though, despite any specific curricular emphasis, programs are focusing on providing a well- 
rounded variety of activities and opportunities that support the physical, social, and cognitive 
development of their student participants. The three main instructional features, which are 
consistently referenced by the research, include (a) the quality of activity implementation, 
(b) offering a variety of activities, and (c) emphasizing principles of youth development. 

Quality of implemeutatiou. According to Yohalem et ah, (2005), setting and 
opportunities provided to participants vary greatly across programs. However, despite the 
variety that exists, there are steps that programs can take during the design of their 
curriculum and implementation of activities to help ensure quality. This is especially 
important for quality-based programs because the tailoring of teaching strategies and 
curricular content to the needs of students may be associated with student outcomes (Bodilly 
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& Beckett, 2005). Employing a variety of research-proven teaching and learning strategies 
can help staff members to increase engagement among students with different learning styles 
(Birmingham et ah, 2005). Furthermore, a failure to design activities that meet the needs and 
interests of students may result in reduced program attendance. For example, Seppanen and 
colleagues (1993) suggested that reduced afterschool enrollment for students in upper 
elementary and above may be the result of a lack of age appropriate activities for older 
students. 

Variety of activities. Providing a variety of activities is a practice supported in the 
afterschool literature. By emphasizing variety, programs are able to extend rather than 
duplicate the school day experience (Wright, 2005; Wright et ah, 2006). This is important 
because afterschool programs that focus rigidly on the school day curriculum have been 
found to have lower participation (Kugler, 2001). In part, this may be due to gender 
differences. For example, Rosenthal and Vandell (1996) found an association between 
participation in programs offering a variety of activities and positive social relationships for 
boys. In addition, their research suggested that a long-term lack of variety in programming 
might be associated with negative outcomes for boys, but not for girls. Posner and Vandell 
(1999) extended this finding when they found gender differences concerning activity 
preferences. In their study, they found that girls spent greater amounts of time socializing and 
doing academic activities during out-of-school time than boys, whereas boys spent greater 
amounts of time than girls participating in coached sports. 

Support youth development. Increasingly, among the educational community, there is 
a call for the development of the whole child. In 2004, the Association for Supervision and 
Curriculum Development (ASCD; 2004) adopted the position that educational practice and 
policy should focus on development of the whole child. As part of this position, they 
provided a framework for how communities, schools, and teachers can contribute to this 
movement. The child development literature also describes a whole child approach to 
cultivate the students’ intellectual, social and emotional well-being in order for them to 
achieve their full potential (Schaps, 2006; Hodgkinson, 2006). 

In order to develop the whole child, education programs need to focus on a variety of 
youth outcomes (American Youth Policy Forum, 2006). As schools are increasingly 
emphasizing cognitive outcomes on core academics, afterschool programs have the 
opportunity to fill an important gap. In other words, afterschool programs can provide 
students with additional opportunities to develop skills, knowledge, resiliency, and self- 
esteem that will help them to succeed in life (American Youth Policy Forum, 2006; Beckett 
et ah, 2001; Huang, 2001; Wright et ah, 2006). With this in mind, researchers and 
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policymakers are placing increasing emphasis on the inclusion of youth development 
principles within afterschool settings (Birmingham et ah, 2005; Kahne et ah, 2001). 

Therefore, the instructional features of afterschool programs should emphasize the 
quality and variety of activities, as well as principles of youth development. This includes 
giving students opportunities to develop personal responsibility, a sense of self-direction, and 
leadership skills (American Youth Policy Forum, 2006; C. S. Mott Foundation, 2005; 
Harvard Family Research Project, 2004; 2005; 2006). 

The purpose of this study is to identify benchmarks and indicators for high quality 
afterschool programs. The research questions for this study are: 

• What are the key benchmarks for quality afterschool programs? 

• What are the key indicators that help define the key benchmarks! 

• How can afterschool programs use a data-based system to identify their own strengths 
and weaknesses, and strive for continuous improvement? 

Identifying quality indicators and benchmarks that are not only preventive of negative 
outcomes, but also promote positive student development will be an important step toward 
informing policy on afterschool activities and instruction. According to the literature 
reviewed, efficient organization, environment, and instruction are crucial for maintaining 
quality in afterschool programs. Mission and vision statements that enable staff to take 
leadership in achieving stated goals and organizing programmatic efforts to achieve those 
goals are essential. Having a strong team of staff members who are qualified, experienced, 
and open to professional development opportunities is critical for successful organization and 
an overall program quality. Beyond program staff, involvement of students’ families and 
communities can enhance the aftersehool program experience, foster program growth, and 
inerease program sustainability. It is also important for quality afterschool programs to 
continuously strive for improvement. Thus, eonsistent and systematic methods of evaluation 
are important to ensure students, families, and communities involved in the program are 
being effectively served. 

Based on these literature reviews, a theoretical model of the indicator system was 
designed. This indieator system focus on four main components of afterschool programs: 
(a) program environment, (b) program organization, (e) instructional features, and 
(d) program self-evaluation and fine tuning of program features. Figure 1 shows the 
theoretical model of the indicator system and its eomponents. 
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This model sets the framework for the development of study instruments and the 
establishment of benehmarks for each of the program component listed in the model. 
In addition, a synthesis of literature on afterschool studies is conducted to extract program 
elements that afterschool experts frequently mentioned as essential features for high- 
functioning, high quality programs. The following section describes the methodology 
employed in this study. 

Methodology 

This section provides detailed descriptions of the study design, synthesis of literature, 
study procedures, and data analysis methodologies. 

Study Design 

This study employed a multi-method approach to address the study questions; 

• What are the core benchmarks for quality afterschool programs? 

• What are the key indicators that help define the core benchmarks for quality 
afterschool programs? 

• How can afterschool programs use a data-based system to identify their own 
strengths and weaknesses and strive for continuous improvements? 

The review of literature provided a theoretical model (Figure 1) in framing the study 
design. Three major program categories were defined; program organization, program 
environment, and instructional features. A comprehensive search and synthesis of literature 
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on quality indicators of afterschool programs was also conducted (see Appendix A for a list 
of literature and frequencies of the benehmarks which surfaced). In order to referenee the 
prevalenee of these indieators as benchmarks for quality programs, instruments (including 
surveys), and interview and observation protoeols were developed by the National Center for 
Research on Evaluation, Standards, and Student Testing (CRESST). 

Next, three well established and high-funetioning aftersehool programs were identified 
through a strategic recommendation proeedure. A referral list was passed among the 
California State-coordinators for them to recommend afterschool programs that they deemed 
as funetioning “above the par.” The most frequently mentioned top five programs were 
identified. A eomprehensive examination on program histories, profiles on parent 
satisfaetions, and performance records was conducted. The records examined affirmed that 
these programs were well-recognized in the field as indieated by their associated field 
records, recognition and awards received, and performance records. Finally, the list of 
programs was presented to the California Poliey Research Center of the University of 
California, and three programs were approved for further investigations. 

Site visits were then conducted at 15 locations for these three programs. The purpose of 
the site visits were two-fold: first, site observations would further eonfirm that these 
programs were in faet high-functioning quality programs (e.g., students were observed to be 
actively engaged and adequately challenged); secondly, in order to establish the indicators 
extracted from the synthesis of literature as benchmarks of high-functioning quality 
programs, the degree of prevalence of the indicators in these programs were examined 
through instruments developed by CRESST. The rationale was that if these indicators were 
first affirmed in the literature by frequeney with which afterschool scholars mentioned them 
as indicators of quality programs, and again confirmed in praetiees of high-functioning 
quality program sites, they could be established as quality benchmarks of afterschool 
programs. 

After the benehmarks were established, our next step was to faeilitate the applieation of 
their usage in assisting policymakers and program funders in decision-making; and to guide 
praetitioners in their own program improvements. Our goal was to create a preliminary tool 
that was user friendly, efficient, and adaptive to different program types. A weighting method 
was employed in developing this tool. This method utilized strategies such as statistical 
weighting.^ Appropriate weights were assigned to all indicators in the process of developing 



^ Most often non-linear regression is done without weighting. The program minimizes the sum-of-squares of the 
vertical distances of the data from the curve. This method gives equal weight to all points, as is appropriate 
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a scale for each benchmark. Following the example of the universal rating scale that most 
testing employs (e.g., a typical score of 70 as a satisfaetory passing seore) a two-thirds rule 
was applied in scoring the benchmarks. The resulted Quality Benehmark Rating System 
could serve as a quiek self-reference guide for afterschool programs. More specifically, this 
Quality Benchmark Rating System was divided into three major categories (program 
organization, program environment, and instruetional features). Each category could yield a 
rating seore that eould serve as a quality index for that benchmark (e.g., as a whole for 
program organization), or be used as a platform to inform programs on their own strengths 
and weaknesses (e.g., based on how they scored under each indicator, they could decide 
where the program needed to focus its future professional development) for continuous 
program improvement. In the following seetions, each study procedure is discussed in detail. 

Synthesis of Literature 

A synthesis of literature was conducted for this study. This approach was similar to a 
meta-analysis of literature, defined as a “type of systematic review that uses statistical 
methods to eombine and summarize the results of several primary studies” (Cook, Mulrow, 
& Haynes, 1997, p. 376). This was the preferred model for analysis in reviewing a large body 
of literature. In this study, the strategy of synthesizing literature was ehosen because few 
studies with qualifying quantitative data or empirical evidences emerged from the literature 
search. In acknowledging the limitations of this process, it is eautioned here that the synthesis 
results were limited in the ability to draw formal inferences to the larger population. 

Literature search. Two literature searches were conducted for this study. The initial 
literature seareh took plaee in December 2006. This formed the basis of the initial theoretical 
model, indicators, and benchmarks. The literature review was further expanded and the 
benchmarks and indicators were revised in May 2007. Searehes were conducted using CSA 
Illumina: ERIC, Education; A Sage Eull-text Collection, NITS, and PsycINFO. Searches 
were conducted using the terms “after sehool program,” “after-school program” or 
“afterschool program” as keywords or descriptors, and with the following parameters: 1985- 
2007, and English-language-only. This identified a total of 1,269 eitations, whieh ineluded 
582 journal artieles, 318 peer-reviewed journals, 115 books, and 62 conference papers. 

The abstracts of the 1,269 citations obtained from the searehes were then carefully 
reviewed and diseussed by the research team members. Titles that indicated the studies 



when you expect experimental scatter to be the same in all parts of the curve. If you expect experimental scatter 
to vary along the curve, you can weight points differentially. 



14 



should be excluded from the synthesis, such as studies of college students, were eliminated. 
Criteria were then established for inclusion into the study: 

• Studies that referred to afterschool programs for K-12 students 

• Studies that either concluded or commented on quality indicators of afterschool 
programs. This means that the study could be either an empirical investigation 
that aimed to identify characteristics of effective afterschool programs, or a 
review of literature that summarized quality indicators based on existing 
literature and/or the author’s own experience and knowledge. 

When abstracts met the criteria mentioned, the research team obtained the full articles. 
These articles were reviewed and discussed by the team members. Out of 1,269 citations, 216 
full articles were reviewed. 

In addition to the databases, the researchers also reviewed the following websites for 
afterschool program evaluation studies and obtained reports on those that were relevant: 
Afterschool Alliance, After School Corporation, Harvard Family Research Project, Rand 
Corporation, and the Private and Public Ventures. 

From all the previously mentioned sources, the research team identified 54 studies that 
met the criteria for inclusion. These studies included review articles, summaries, policy 
reports, and evaluation reports, and were often written by researchers and experts who had 
extensive experience in the field of afterschool programming. Each of the 54 studies was 
coded for information regarding benchmarks/indicators of quality for afterschool programs. 
After carefully coding the quality indicators and engaging in extensive debate and deliberate 
discussions, 13 benchmarks were extracted. They were: (a) safety, (b) health, (c) physical 
resources, (d) human relationships, (e) programming/activities, (f) family/community 
involvements, (g) staffing/professional development, (h) academic support, (i) social 
development, (j) enrichments, (k) positive youth development, (1) evaluation, and (m) 
management/administration. For details of the literature reviewed, please see Appendix A. 

Despite observed differences among the 54 articles, there were substantial overlapping 
consistencies in opinions. Each benchmark received support from at least one-quarter of the 
sources. Benchmarks that received the strongest levels of support (at least 75% or 40 
sources) all focused on issues of program environment: Staffing/Professional Development, 
Programming/Activities, and Management/ Administration. In contrast, those that received 
moderate support (less than 50% or 27 sources) were distributed across the broad categories 
like safety, human relationships, and health. Positive youth development, a relatively recent 
focus in the afterschool curriculum, was mentioned the least at the time the literature were 
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reviewed (15 sources), following by social development (18 sources), evaluation (19 
sources), and physical space/resources (21 sources). 

Following the structure of the theoretical model, the 13 benchmarks were then grouped 
under the three broad categories of program organization, program environment, and 
instructional features, as shown in Table 1. 

Table 1 



Distribution of Benchmarks Across Categories 



Program organization 


Program environment 


Instructional features 


Safety 


Programming/Activities 


Academic Support 


Health 


Family/Community Involvement 


Social Development 


Physical space/resources 


Staff/Professional Development 


Enrichment 


Human relationships 


Management/Administration 

Evaluation 


Positive Youth Development 



Instrument Development 

Survey and observation protocols were developed by the National Center for Research 
on Evaluation, Standards, and Student Testing (CRESST) to examine the identified quality 
indicators at high functioning afterschool program sites in Los Angeles County. All items for 
the protocols were either extracted from the literature synthesis or adapted from existing 
instruments for the evaluation of afterschool programs.^ A description of each instrument is 
provided in the following text. 

Observation protocol. The observation protocol was designed to examine quality 
indicators of program environment and instructional features. The observation protocol 
focused on the examination of instructional methods and strategies; academics and 
enrichment provided; personal responsibility, self-direction, and leadership of staff; 
collaboration with day schools; safety, health, and physical space; code of conduct; 
relationships; and overall program climate (see Appendix B). 

Site staff survey. The site staff survey was designed to examine indicators in all three 
components of program organization, program environment, and instructional features. The 
survey included items on staff support, health and safety, indoor space, staff and student 

^ Examples such as the New York State Afterschool Network (NYSAN) Program Quality Self-Assessment 
Tool, the National Association of Elementary School Principals (NAESP) Checklist for Quality Indicators, and 
the Safe and Sound Workbook, and instruments developed and validated in previous CRESST studies. 
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relationships, activities and goals, academics and enrichment, as well as having a code of 
conduct (see Appendix C). 

Program director survey. The program director survey was designed to examine 
indicators of program organization. The program director survey included items on 
management, administration, staff experience and training, family involvement, community 
partnerships, and evaluation (see Appendix D). 

Program Identiflcation and Recruitment of Participants 

As a result of the strategic afterschool program search, three well established and well- 
referenced afterschool programs in Los Angeles County were identified. These programs 
were: Los Angeles Better Educated Students for Tomorrow (LA’s BEST), Lawndale 
Realizing Amazing Potential (RAP), and Pasadena Leading Educational Achievement — 
Revitalizing Neighborhoods (LEARNs)."^ 

These three programs were considered appropriate candidates for this study because of 
their reputations within the afterschool community, the similarity in student populations that 
they serve, and the evidences of high-functioning records they produced (such as external 
evaluation reports; testimonies of parent satisfaction; perceived improvements on student 
performances as referred by school teachers, program staff, and parents). Each has been 
designated as a California After-School Partnership (CASP) Regional Learning Center, and 
LA’s BEST was selected by World Hunger Year (WHY) as one of the top afterschool 
programs in the State of California. All three programs serve similar student populations and 
implement a specific set of goals that guide their programmatic efforts to provide quality 
afterschool care. Each individual site included in the study provided a program structure and 
instructional features that met the stated goal of their afterschool program. Although the 
individual sites varied in structure, all three afterschool programs had a clear emphasis on 
academic enrichment balanced with physical activities, performing and visual arts, and 
cultural activities. The student population at all three programs were predominately Eatina/o, 
followed by African American students. Furthermore, White, Asian, Native American, and 
Pacific Islander students composed approximately 15% of the total population served at each 
program. On average, most of the students enrolled in the programs qualified for free or 
reduced lunch (see Appendix E for more detailed individual program descriptions). Table 2 
shows the specific number of participants who were recruited at each afterschool program. 



Although Pasadena LEARNs and Lawndale RAP offer middle school and/or high school components, data 
collection was limited to program sites geared toward elementary school students. 
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Table 2 



Study Participants by Role and Afterschool Program 



Participant 


Survey 


Program Coordinators (Total) 


17 


LA’s BEST 


5 


Lawndale RAP 


5 


Pasadena LEARNs 


7 


Site Staff (Total) 


102 


LA’s BEST 


39 


Lawndale RAP 


26 


Pasadena LEARNs 


37 



Note. LA’s BEST = Los Angeles Better Educated Students for Tomorrow, 
RAP = Realizing Amazing Potential, LEARN s = Leading Educational 
Achievement — Revitalizing Neighborhoods. 



Data Collection Procedures 

Approval was obtained from the UCLA Offiee for the Proteetion of Human Researeh 
Subjects concerning the appropriateness of the study procedures and instruments on April 13, 
2007. Appropriate permissions and consent forms were obtained from all study subjects. The 
operation offices at each of the three programs recommended five individual sites from their 
programs for site visits. 

Survey administration. Site staff members and program directors were each surveyed 
once during the period of study. Survey instruments were mailed to the sites along with the 
staff information sheets used for consent. The instruments were completed by site staff and 
program directors during the operation of the afterschool program and returned to the 
CRESST researchers at the time of the site visits. 

Observations, Four observations (two at a time) were conducted at each of the 15 
study sites during 2007. After coordinating with the program directors, the study researchers 
visited each of the afterschool sites and observed different grade levels and different 
activities offered. Observations of the different programs were conducted at three different 
times during the school year: LA’s BEST during spring 2007, Lawndale RAP during summer 
2007, and Pasadena LEARNs during fall 2007. 

Data Analysis 

Descriptive statistics were employed in analyzing the observation and survey data. 
A weighting system was developed to examine whether certain indicators were prevalent at 
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the afterschool sites and to determine which indicators should be considered core benchmark 
components. 

Step 1 - Establishing the two-thirds rule. The primary objective of this study was to 
identify key benchmarks for quality afterschool programs, and secondarily to design a 
preliminary system for the application of these benchmarks in determining program quality. 
For ease of use, this study designed a rating tool that was user friendly, efficient, and 
adaptive to different afterschool program types; in order to provide a point of reference in 
scoring the indicators and benchmarks, a two-thirds rule was applied. This rule was 
established following the universal example of a rating scale that most curriculum scoring 
employs (i.e., 70% correct is usually considered as a satisfactory passing score). 

The two-thirds rule was used to make determinations as to whether the indicators 
identified were prevalent at the afterschool sites visited. Data were analyzed at the site level. 
If an indicator was examined by a single item from the instruments, then two-thirds of the 
responses aggregated to the site level would be required in order to consider the indicator as 
prevalent or “met.” If an indicator was examined using multiple items from the instruments, 
then two-thirds of the responses, or at least two-thirds of those items aggregated to the site 
level would be required to consider the indicator as “met.” For example, if a site with 10 staff 
members responded to a benchmark instrument with 4 indicators or items, and the responses 
from the site staff were 6 positive responses to item one, 6 positive responses to item two, 
8 positive responses to item three, and 9 positive responses to item four; this site would fail 
to meet the benchmark based on the two-thirds rule (only 2 items have passed the two-thirds 
rule). If, however, the site had 6, 7, 8, and 9 positive responses to the four respective items, 
the site would be considered to have met the benchmark because three of the four items 
would have passed the two-thirds rule. All items or indicators under the benchmark would 
then be accepted into the Quality Benchmark Rating System. This same procedure was 
applied to all instruments. When an indicator was analyzed with items from different surveys 
(staff surveys, project director surveys), aggregate responses were weighted equally. When 
an indicator was analyzed with items from survey and observation protocol, aggregate survey 
responses were weighted more heavily (2:1) than the observers’ responses. 

Step 2 - Establishing the weighting system. In applying the indicators extracted from 
the literature to the Quality Benchmark Rating System, a weighting method was used. This 
method assumed strategies as used in statistical weighting. This allowed the study to give 
equal weight to all extracted indicators, as appropriate to their importance referenced in the 
literature. 
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Each benchmark was assigned a maximum of 10 points (see Tables 21-23). The 
indicators within each benchmark were discussed and weighted by the research team based 
on the team’s understanding of their relative importanee after conducting the synthesis of 
literature. More speeifieally, the weight for eaeh indicator was established by revisiting the 
literature synthesis (see Appendix A). The researeh team reviewed all the literature under 
each individual indicator. A checklist or numerical scoring tablet was created for each 
benchmark with all the indieators for that benchmark listed underneath. Each time an 
indieator was mentioned in the literature a eheck was marked under the appropriate indicator. 
When all the literature under the benehmark was re-examined, a numerieal seore was 
calculated for eaeh indicator. The pereentage of the frequeney of eaeh indieator mentioned 
was then rounded to the first decimal as the preliminary resulted seore for that indieator. 
Weekly researeh team diseussions on the appropriateness of assigned weight to the rating 
score were eondueted until consensus had been reaehed on all items. 

To provide a conerete example on the weighting system, if there were three indieators 
for a given benchmark, and the third indieator was considered slightly more important than 
the first two indieators, the assigned weights for the three indieators might be set at 3, 3, and 
4, respectively. If a site was determined to have met all three indicators they would reeeive a 
score of 10 points (3 + 3+4) for that benehmark. Likewise, if a site was determined to have 
met only the first and third indieators, they would reeeive a seore of 7 points (3 + 0 + 4) for 
that benc hm ark. 

The following seetion deseribes the proeedures used to establish the eore benehmarks 
for high funetioning quality programs. Given that all benchmarks and indicators were 
extracted from the synthesis of literature, these indieators should be eonsidered as researeh 
results of what high quality aftersehool programs {under ideal conditions) ought to have in 
plaee. However, in daily praetiees, the afterschool field conditions might not have always 
been 'ideal/ and not all afterschool programs had the same emphasis. In fact, afterschool 
practitioners constantly had to juggle with multiple daily demands, limited time, spaee, and 
resources; they may not have been able to practice all the key benchmarks and indicators for 
high quality programs. At the same time, afterschool programs that focused on developing 
the academie skills of their students may not have been as concerned about the “eharaeter 
development” of their students as programs that foeus on social development might be. 
Under this rationale, the two-thirds rule was established: that is, if the benchmarks were 
confirmed to be prevalent at most of these high-funetioning sites (two-thirds), they were 
considered as neeessary eomponents for the daily operation of quality aftersehool programs. 
In another words, these were indieators that were prevalent in most of the high-functioning 
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sites studied, and therefore, eonsidered neeessary conditions for afterschool programs to 
ensure program quality. Under the same rationale, the benchmarks that received a score 
below 7 were benchmarks that were confirmed by the literature as important indicators of 
quality program components that could lead to positive student outcomes, but that failed to 
be practiced regularly at these high functioning programs. These indicators were thus defined 
as additional or “exemplary” components that programs could use to further enhance their 
program quality. 

Each indicator was also examined individually. As a result, a benchmark may have had 
two indicators that passed the two-thirds rule, but one that did not. Using the same rationale 
that these were indicators identified in the literature, as long as the benchmark as a whole 
passed the two-thirds rule, all indicators were accepted under the benchmark, but given 
different weight as described in the analysis section. 

Utilizing the results of the weighting system, means were calculated for each 
benchmark under program organizations. These means represented the aggregated results 
across the 15 afterschool sites in the study. As discussed previously, benchmarks that 
received a mean score of 7 out of 10 were considered as “core” components of quality 
afterschool programs. Benchmarks that received a mean score lower than 7 were considered 
as “additional” or “exemplary” benchmarks that afterschool programs could use to enhance 
their program quality. 

Establishing the Core Benchmarks 

Under the performance of the weighting system, means were calculated to determine 
which benchmarks should be considered as core components of quality afterschool programs. 
These determinations were made by aggregating the results across the 15 sites. Following 
previous examples, a threshold of 70%, or just over two-thirds, was established to determine 
if a benchmark should be considered a core component. In this case, a mean score of 7 across 
the program sites was required to establish a benchmark as core. The following section 
describes the analysis and the procedures used in establishing the prevalence of the quality 
indicators. 

Formulating the “Quality Indicator System” 

In order to provide more clarity to the construction of the “Quality Indicator System,” 
the analysis was presented under the three major components of afterschool effectiveness: 
program organization, program structure, and instructional features. 
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Program Organization 

For program organization, seven benchmarks were extracted from the synthesis of 
literature: (a) program management, (b) administration, (c) staff support, (d) staff training, 
(e) family involvement, (f) community partnership, and (d) program evaluation. Figure 2 
illustrates the structure of this benchmark. 



Program Organization 




Management 



Staff Support Family Involvement Community Partnership Evaluation 



Clear mission 
statement 

Staff input 

Student input 

Collaboration 
with day school 



Administration 



Staff-student ratio 

Staff competency 

Staff/Director 

Professional 

development 

Cultural diversity 



Staff Training 





r 


Program 




Staff salary 


policies 










Staff feedback 


Maintain 






budget 




staff 






orientation 


Sustaining 







Communication 
with parents 



Parent 

involvement 

Parent feedback 



^ \ 

Partnership with 

communities 

Service projects 

Plan for 
community 
involvement 



Evaluation of 
staff 

performance 

Program 

activities 

Student 

engagement 

Student 

outcomes 

Continuous 

improvement 



Figure 2. Indicators for Program organization. 

Program management. Program management describes the capacity of a program to 
have “a collaborative management system to meet specific goals outlined in the mission 
statement.” The four indicators for this benchmark included: 1) Considers staff input in 
decision-making; 2) Considers student input in decision-making; 3) Clear mission statement 
is present; and 4) There is day school and afterschool collaboration. Among the strongest 
indicators for determining quality program management were having a clear program 
mission statement (M = 1.00) and the incorporation of staff in program decision-making 
(M= 0.93). In other words, all 15 sites included in this study indicated that their program had 
a clear mission statement, and staff inputs were considered strongly in program decision- 
making. Student input in decision-making was also strong (M = 0.80), with the indicator 
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being met at 12 sites. In contrast, only 10 sites (M= 0.67) indicated that they collaborated 
with the day school at their sites. All indicators were accepted as core benchmarks. 

Please note here that once a benchmark was established as core, all the indicators under 
the benchmark were accepted as well. It was beyond the scope of this study to conduct any 
further examination. Future studies should make use of the preliminary analysis on the 
indicators and conduct further testing of the instrument to make refinements/attunements on 
the selection of indicators according to the study results. 

Program administration. Program administration was defined as having “effective 
management and plan for long-term sustainability and growth.” The three indicators for this 
benchmark were: 1) Develops program policies for student participation and attendance; 
2) Budget is maintained and adjusted to meet resource needs; and 3) Long-term financial 
plan in place for sustaining and fostering program growth. The strongest indicator measured 
for this benchmark was developing program policies for student participation and attendance, 
with all 15 sites reporting that these policies were in place at their programs (M= 1.00). Most 
program directors also reported that they had established long-term financial plans to 
maintain program quality and sustainability as outlined in the third indicator (M= 0.80). 

Maintaining a program budget to meet resource needs was the only indicator for the 
program administration benchmark that did not meet the two-thirds rule when aggregated 
across the programs (M= 0.60). As shown in Table 3, this indicator was analyzed with one 
item from the program director survey, which received a mean of 63.33. Furthermore, the 
standard deviation was high {SD = 48.00) indicating that maintaining and adjusting the 
budget was not a consistent goal across the 15 sites included in this analysis. 

Table 3 



Descriptive Statistics for Indicator 2 of the Program Administration Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Program Director 












Program budget periodically adjusted 
based on resource needs 


15 


63.33 


48.06 


0.00 


100.00 



Staff Support. The three indicators for this benchmark included: 1) Staff is well-paid; 
2) Staff are provided performance feedback; and 3) Staff receives an orientation before 
working with youth. Indicators 2 and 3 were both strongly present across the sites (M= 0.87 
and M = 0.93, respectively). Program staff reported that they received an orientation to 
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review their job responsibilities before working with students and were provided with 
performance feedbacks. 

Staff being paid well was the only weak indicator for the staff support benchmark 
(M = 0.40). Two items from the staff survey were used to measure this indicator. Table 4 
shows that although the mean for the item regarding salary structure was strong (M= 80.89), 
the mean for the second item regarding being paid well was low (M= 60.30). This shows that 
although a clear salary structure was in place at most of the sites, the indicator failed to be 
met because staff at many of the sites did not feel that the compensation was adequate for 
the work. 



Table 4 

Descriptive Statistics for Indicator 1 of the Staff Support Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Site Staff 












Clear salary structure in place for staff 


15 


80.89 


18.77 


33.30 


100.00 


Staff are paid well for the work they 
do in the program 


15 


60.30 


28.68 


0.00 


100.00 



Staff Experience and Training. The fourth benchmark for program organization was 
defined as “all staff members have adequate training and experience to ensure high quality 
instruction.” Five indicators were used to measure this benchmark. The indicators were; 
1) There is an adequate staff- to- student ratio; 2) Staff is competent in core academic areas; 
3) Staff participates in professional development; 4) Program director participates in 
professional development; and 5) Staff reflects the cultural diversity of the community. The 
majority of the indicators for this benchmark met the two-thirds rule. The strongest indictor 
for this benchmark was Indicator 5, concerning whether the program staff reflected the 
cultural diversity of the community (M = 1.00). Indicators 2 and 4 were equally strong, 
having the same mean score of (M= 0.93). Indicator 3 was also present at a majority of the 
sites (M= 0.80). 

Of the five indicators for the staff experience and training benchmark, only the staff-to- 
student ratio indicator failed to hold up to the two-thirds rule (M= 0.53). Table 5 shows that 
this indicator was measured by one item from the site staff survey and one item from the 
program director survey. Although it appeared that both means were fairly high (M= 76.49 
and M = 93.33, respectively), the standard deviations and minimum scores showed that 
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variability existed aeross the sites. In other words, many site staff felt the staff-to-student 
ratio was not adequate at their respective sites. In contrast, most program directors felt 
confident that they had enough qualified staff to provide high quality instruction. These 
findings suggest a possible conflict between site staff and program directors in regards to 
their perceptions on the adequacy of the staff-to-student ratio in meeting staff and student 
needs. 

Table 5 



Descriptive Statistics for Indicator 1 of the Staff Experience and Training Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Site Staff 












With the current staff-to-student 
ratio, staff are able to give 
sufficient attention to all students 


15 


74.69 


24.20 


33.30 


100.00 


Program director 












Enough qualified instructors on 
staff to ensure high quality 
instruction 


15 


93.33 


25.82 


0.00 


100.00 



Family involvement. Overall, the indicators for family and community involvement 
were considered weak. Family involvement was defined as a program that “has a clear plan 
for family involvement.” The three indicators for this benchmark included: 1) Staff regularly 
communicates with parents/families; 2) There is a program plan in place for parent 
involvement; and 3) Provides parents with opportunities to provide feedback about the 
program. Only one of the three indicators for this benchmark met the two-thirds rule. 
Examining these indicators at site level showed that Indicator 1, concerning staff and parent 
communication was prevalent at 14 sites (M = 0.93). In contrast, the remaining two 
indicators were only present at 7 sites (M= 0.47). 

Table 6 illustrates the means of the items under Indicator 2. The table illustrates a 
moderate mean for the item: families being welcomed to visit the program at anytime 
(M= 73.33). There was a higher mean for the item: encouraging families to be involved in 
the afterschool program (M = 86.67), but the remaining items had low and very low mean 
scores (M= 53.33 and M = 32.14, respectively). Additionally, very few program directors 
felt that a clear plan was established to encourage parent involvement at their sites, and even 
fewer felt that their programs provided learning opportunities to parents. Subsequently, this 
indicator failed to meet the two-thirds rule. 
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Table 6 

Descriptive Statistics for Indicator 2 of the Family Involvement Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Program Director 












Clearly defined plan for program 
involvement 


15 


53.33 


51.64 


0.00 


100.00 


Family is welcome to visit anytime 


15 


73.33 


45.77 


0.00 


100.00 


Staff encourages families to get 
involved 


15 


86.67 


35.19 


0.00 


100.00 


Learning opportunities for parents 


15 


32.14 


46.44 


0.00 


100.00 



Table 7 illustrates the means for the items used to examine indieator 3. The table shows 
that two items were used from the program director survey. Although the item concerning 
parent input in decisions about programming resulted in a moderate mean (M = 70.00), the 
mean for the second item regarding whether parents are surveyed about their satisfaction was 
low (M = 60.00). Furthermore, both items had high standard deviations {SD = 45.51 and 
SD = 50.71, respectively) indicating a lot of variability among sites. This indicator also failed 
to meet the two-thirds rule. 

Table 7 



Descriptive Statistics for Indicator 3 of the Family Involvement Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Program Director 












Parent input is considered in decisions 
about programming 


15 


70.00 


45.51 


0.00 


100.00 


Parents are surveyed to measure their 
satisfaction 


15 


60.00 


50.71 


0.00 


100.00 



Community partnerships. Three indicators were used to measure this benchmark: 
1) There are established partnerships with local community organizations; 2) Program 
encourages student to participate in service projects and programs; and 3) There is a plan in 
place for community involvement. Similar to the benchmark concerning family 
involvements, the indicators for community partnerships were generally weak. The only 
indicator that met the two-thirds rule was youth encouragement to participate in service 
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projects and programs (M= 0.73). Indicators 1 and 3 had means lower than 0.67 (M= 0.60 
and M= 0.46, respectively). 

As shown in Table 8, indicator 1 was measured by one item from the program director 
survey. The table shows that this item received a mean of (M= 63.33). Furthermore, there 
was a high standard deviation of 48.06 showing variability across the sites. In other words, 
establishing partnerships with local community organizations was not a priority at many of 
the sites included in this analysis. 



Table 8 

Descriptive Statistics for Indicator 1 of the Community Partnerships Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Program director 












Have established partnerships with 
local community orgs. 


15 


63.33 


48.06 


0.00 


100.00 



As for indicator 3 (see Table 9), three of the four items from the program director 
survey received moderate to high means (M= 73.33 and above). The only item that received 
a low mean was: “bringing in guest speakers” (M = 36.67). It should be noted that all four 
items had large standard deviations, ranging from 37.25 to 48.06, indicating that program 
sites differed greatly in how they planned for community involvement. 



Table 9 

Descriptive Statistics for Indicator 3 of the Community Partnerships Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Program director 












Qualified volunteers are recruited to 
work in the afterschool program. 


15 


73.33 


45.77 


0.00 


100.00 


We involve local community 
partnerships in program planning. 


15 


73.33 


41.69 


0.00 


100.00 


We regularly bring in community 
members as guest speakers. 


15 


36.67 


48.06 


0.00 


100.00 


We recruit volunteers from the 
community. 


15 


82.14 


37.25 


0.00 


100.00 



27 



Evaluation. The final benchmark under program organization examined whether the 
“program has a system in place for evaluation of students, staff, parents, and program 
activities.” There were five indicators for this benchmark including; 1) Method of evaluation 
for staff performance; 2) Method of evaluation for program activities; 3) Method of 
evaluation for student engagement; 4) Evaluation of students’ improvement on academic 
and/or social skills; and 5) Evaluation findings are used for program improvement. Most of 
the indicators met the two-thirds rule. Indicators 1 and 2 each had a mean of 1.00, signifying 
that all 15 sites had an evaluation method in place for staff and program activities. 
Furthermore, indicators 3 and 5 were also strong, with both having a mean of 0.80. 

Evaluating students’ academic and/or social skills improvement was the only indicator 
that did not meet the two-thirds rule for this benchmark (M = 0.53). Table 10 shows the 
results of the two items under this indicator (from the program director survey). The table 
shows that the means for both academic and social improvement were moderate (M= 70.00 
and M= 71.43, respectively). However, the standard deviations were high (ranging from 
42.58 to 45.51) indicating a great deal of variability across the 15 sites. 

Table 10 



Descriptive Statistics for Indicator 4 of the Evaluation Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Program director 












Students’ academic improvement 


15 


70.00 


45.51 


0.00 


100.00 


Students’ social skills development 


14 


71.43 


42.58 


0.00 


100.00 



Setting up the System for Program Organization, Given that all indicators were 
extracted from the synthesis of literature, these indicators should be considered as research 
results of what high quality afterschool programs {under ideal conditions) ought to have in 
place. However, in daily practices, the afterschool field conditions may not have always been 
“/c/ea/.” In fact, afterschool practitioners constantly had to juggle with multiple daily 
demands, limited time, space, and resources; they may not have been able to practice all the 
key indicators for high quality programs. Linder this rationale, the two-thirds rule was 
established: that if the benchmarks were confirmed to be prevalent at most of these high- 
functioning sites (two-thirds), they were considered as necessary components for the daily 
operation of quality afterschool programs. In another words, these were indicators that were 
prevalent in most of these high-functioning sites, and therefore, considered necessary 
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conditions for afterschool programs to ensure program quality. Under the same rationale, the 
benehmarks that reeeived a seore below 7 were benehmarks that were eonfirmed by literature 
as important indicators of quality program components that eould lead to positive student 
outcomes, but failed to be practieed regularly at these high-functioning programs. These 
indieators were thus defined as additional, or “exemplary” components that programs can use 
to further enhance their program quality. 

Utilizing the results of the weighting system, means were caleulated for each 
benchmark under program organizations. These means represented the aggregated results 
across the 15 aftersehool sites in the study. As discussed previously, benehmarks that 
received a mean seore of 7 out of 10 were considered as “eore” eomponents of quality 
afterschool programs. Benchmarks that received a mean score lower than 7 were eonsidered 
as “additional” or “exemplary” benchmarks that afterschool programs could use to enhanee 
their program quality. 

This categorization proeedure is illustrated in Table 11, showing the means and the 
strategies employed to eategorize the program organization benehmarks as core or 
exemplary. 



Table 1 1 

Program Organization - Mean Scores, Core and High Quality Benchmarks 



Benchmark 


Mean score 


Core benchmark 


Exemplary benchmark 


1 . Program Management 


8.50 


P 




2. Program Administration 


8.00 


P 




3. Staff Support 


8.00 


P 




4. Staff Experience and Training 


8.40 


P 




5. Family Involvement 


7.00 




P 


6. Community Partnerships 


5.73 




P 


7. Evaluation 


8.27 


P 





As shown in Table 11, five of the seven benchmarks under program organization were 
established as core eomponents. The most prevalent benchmarks (as indicated by the mean 
scores) were program management (M = 8.50), followed by staff experience and training 
(M = 5.40), and evaluation (M = 8.27). The program administration and staff support 
benchmarks both had equally strong means (M = 8.00). The benchmark for family 
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involvement just barely met the threshold of 70%, with a mean of exaetly 7.00. Beeause the 
other benehmarks were rated considerably higher, the benchmark of family involvement was 
grouped under the exemplary category. The only benchmark that scored below .70 was 
community partnerships (M= 5.73), which was also grouped under the exemplary category. 

Program Environment 

Under the framework of the literature synthesis, program environment was represented 
by four benchmarks: safe environment, student health and well-being, well equipped and 
suitable physical spaces, and positive relations. Figure 3 illustrates the outline of this 
program component. 



Program Environment 




Safety Health Physieal Spaee P ositive Relationsh ips 



Clean & secure 




Promote healthy 
habits 




Sufficient space 




Staff: student 
relationship 


Prevention 

strategies 




Nutritious snacks 




Proper layout of 
space 




Student: student 
relationship 


Proper 

supervision 




Minimize health 
risks 

Safe equipments 




Provision for 

multiple 

activities 




Staff: staff 
relationship 



Figure 3. Indicators for program environment. 



Safe environment. Safe environment was defined as “a space that is safe, clean, and 
secure.” This benchmark had three indicators, which included: 1) Program space that is safe, 
clean and secure; 2) System is in place that keeps unauthorized people from taking students 
from the program; and 3) Students are carefully supervised. Indicators 2 and 3 were very 
strong, each having a mean score of 1.00. The indicator regarding having a safe, clean, and 
secure environment also had a very high mean score of 0.93. This suggests that having a 
system in place to keep unauthorized people from taking students and ensuring that students 
were well supervised were considered as priorities at all 1 5 sites. 

Student health and well-being. Student health and well-being was defined as the 
“program environment should enhance students’ health.” The indicators for this benchmark 
were: 1) Program environment should enhance students’ health; 2) Provides healthy and 
nutritious snacks; 3) Staff work hard to minimize health risks; and 4) Equipment is safe for 
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activities and play. The majority of indicators for this benchmark met the two-thirds rule. 
Enhancing students’ health and having safe equipment were strongly present across the sites 
(M= 0.93 and M= 1.00, respectively). The indicator concerning the providing of nutritious 
snacks was also moderately present (M= 0.73). 

Staff minimizing health risks was the only weak indicator for the student health and 
well-being benchmark (M= 0.20). One item from the site staff survey and one item from the 
observation protocol were used to examine this indictor. As mentioned previously, the staff 
responses and observation responses were weighted 2:1. Table 12 shows that although the 
mean for the item concerning the minimizing of health risks was moderate (M= 69.44), the 
mean for the item in regards to attending to students’ allergy and medical needs was low 
(M= 57.00). Furthermore, the standard deviations were moderate for both items {SD = 24.16 
and SD = 30.48, respectively), indicating variability across the sites. 

Table 12 



Descriptive Statistics for Indicator 3 of the Student Health and Well-Being Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Site Staff 

Allergy and medical needs provided to staff 


15 


57.00 


24.16 


16.70 


90.00 


Observation 

Staff take steps to minimize health risks 


15 


69.44 


30.48 


0.00 


100.00 



Well-equipped and suitable physical space. The fourth benchmark for program 
environment was defined as “the provision of physical space that is appropriately equipped 
and suitable for afterschool.” There were three indicators for this benchmark: 1) The 
program’s indoor and outdoor space meets the needs of all program activities; 2) Space is 
arranged well for a range of activities; and 3) Space is arranged well for simultaneous 
activities. All three indicators met the two-thirds rule. Indicator 1 regarding sufficient space 
in meeting the students’ needs for all activities was the strongest and was present at all 15 
sites (M = 1.00). Indicators 2 and 3 were also very strong (M = 0.93 and M = 0.87, 
respectively), suggesting that most of the sites had spaces for a range of activities and 
simultaneous activities. 

Positive relationships. The final benchmark for program environment was defined as a 
“program that develops, nurtures, and maintains positive relationships.” This benchmark 
included 10 indicators sparming the following three subcategories: (a) staff-student 
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relationships; (b) student-student relationships; and (c) staff-staff relationships. Results for 
each of the subcategories will be presented separately. 

The six indicators for the staff-student relationships subcategory included; 1) Small 
staff-to-student ratio; 2) Program has guidelines about staff-student expectations; 3) Staff 
members relate to students in positive ways; 4) Staff members respond appropriately to the 
individual needs of the students; 5) Staff members encourage students to become more 
responsible; and 6) Staff members interact with students to help them learn. All six indicators 
concerning the staff and students met the two-thirds rule when aggregated across the program 
sites. More specifically, indicators 1, 2, 4, and 6 were present at all of the sites (M= 1.00). 
The indicator concerning whether staff members relate positively to students also resulted in 
a very high mean (M = 0.87). Indicator 5 was also moderately present (M = 0.73), which 
implies that the staff encouraged students to act responsibly. 

The seventh indicator for the positive relationships benchmark was part of the student- 
student subcategory: 7) Students interact with one another in positive ways. This indicator 
was strong with a mean of 1.00, demonstrating that students were interacting in a positive 
manner with one another at all 15 program sites. 

The final subcategory for the positive relationships benchmark focused on staff-staff 
relationships. The three indicators for this subcategory included; 8) Staff members work well 
together to meet the needs of students; 9) Staff members communicate with each other while 
the program is in session; and 10) Staff members provide role models of positive adult 
relationships. All three indicators were strong to very strong, resulting in means of 1.00, 0.93, 
and 0.80, respectively. In other words, all of the staff-staff relationship indicators met the 
two-thirds rule for establishment as core benchmarks. 

Setting up the System for Program Environment, Similar to establishing the core 
benchmarks for program organization, the results of the weighting system and calculated 
means were used to make the final determination. The means represented the aggregated 
results across the 15 afterschool sites. Benchmarks that resulted in a mean score of 7 or 
higher were considered as core components for quality afterschool programs, whereas those 
that received a score below 7 were defined as “exemplary” components that could further 
enhance program quality. Table 13 shows the means and categorizations of benchmarks for 
program environment. 
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Table 13 



Program Environment - Mean Scores, Core, and High Quality Benchmarks 



Benchmarks 


Mean score 


Core 

benchmarks 


High quality 
benchmarks 


8. Safe Environment 


9.73 


P 




9. Student Health and Well-Being 


6.60 




P 


10. Well-Equipped and Suitable Physical Space 


9.33 


P 




1 1 . Positive Relationships 


9.37 


P 





As shown in Table 13, all but one of the benchmarks was established as a core 
component for program environment. The most prevalent benchmarks (as indicated by the 
mean scores) were safe environment (M = 9.73), followed by positive relationships 
(M= 9.37), and well equipped and suitable physical space (M= 9.33). The only benehmark 
that eould not be established as core was the benchmark for student health and well-being 
(M=6.60). 

Instructional Features 

Instructional features were represented by three benchmarks: quality of 

implementation, variety of activities, and supporting youth development. Figure 4 illustrates 
the benchmarks and indicators for this program component. 

Instructional Features 




Quality of Implementation Variety of Activities Support Youth Development 



Appropriate activities 




Academics 




Youth development 


Student engagement 
Cultural diversity 




Enrichment 




Personal responsibility 


Meets students’ needs 








Self-direction 


Teaching & learning 

Opportunities for 
practice 




Socialization 




Leadership 









Figure 4. Indicators for Instructional Features. 
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Quality of implementation. Quality of implication was defined as the “program 
provides a variety of age-appropriate aetivities that refleet the goals and philosophy of the 
program.” The six indieators for this benchmark included: 1) Aetivities are appropriate 
(i.e., ages, learning styles, and abilities) for students in the program; 2) Aetivities are in line 
with the interests of the students in the program; 3) Activities reflect the languages and 
eultures of the families served; 4) Activities meet the physical, social, and emotional needs of 
the students; 5) Program uses a variety of instruetional methods and strategies that reflect 
current researeh and policies on teaehing and learning; and 6) Program offers multiple 
opportunities for students to develop and praetiee new skills. All six indicators met the two- 
thirds rule. Five indicators were very strong, with eaeh being prevalent at all 15 sites 
(M= 1.00). The indieator eoneeming whether aetivities reflect the languages and eulture of 
their student population was also very strong, and resulted in a mean of 0.93. This implies 
that issues of appropriateness and pedagogy of teaehing methods were taken into serious 
eonsideration at all 15 sites. 

Variety of activities. The seeond benehmark for instruetional features deseribed 
whether aftersehool programs provided “a balance between aeademies and enriehment.” This 
benehmark ineluded live indieators spanning the following three subcategories: (a) core 
aeademies, (b) enriehment, and (e) soeialization. Results for eaeh of the subeategories will be 
presented separately. 

The eore aeademies subeategory includes the indieators of: 1) Offers high quality 
aeademie support, sueh as tutoring and homework help; and 2) Offers instruction in a variety 
of core aeademie areas. Only one of the indieators for this subeategory met the two-thirds 
rule for establishing eore benehmarks. Indieator 2, eoneeming having aetivities that focus on 
core academies, was present at all 15 sites (M= 1.00). In eontrast, the indieator regarding the 
offering of tutoring and/or homework was only present at 9 sites (M= 0.60). 

Table 14 illustrates the detailed results for examining indieator 1 (regarding tutoring 
and homework help). Two items from the site staff survey and one item from the observation 
protoeol were used. The site staff items were weighted more heavily than observation items 
(2:1). Table 14 shows that although the mean for the site staff item concerning homework 
was strong (M = 98.43), the mean for the seeond site staff item eoneeming tutoring was 
moderate (M = 70.95). Likewise, the mean for the observation item eoneeming homework 
was moderate (M= 70.00); Furthermore, both of the items with lower means had moderate to 
high standard deviations, indieating variability aeross the program sites. As a result, only the 
site staff homework item met the two-thirds threshold. It should also be noted that the mean 
for the observation item may have been affeeted by the timing of data collection. More 
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precisely, some observations took place during summer intersession or on a Friday; times 
when elementary students were generally not assigned homework by their day school 
teachers. It should also be noted that although most afterschool programs provided 
homework help, not many programs conducted serious tutoring as part of their program. 

Table 14 



Descriptive Statistics for Indicator 1 of the Variety of Activities Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Site Staff 


Homework 


15 


98.43 


4.16 


87.50 


100.00 


Tutoring 


15 


70.95 


22.58 


28.60 


100.00 


Observation 


Homework 


15 


70.00 


42.47 


0.00 


100.00 



The third and fourth indicators for the variety of activities benchmark were part of the 
enrichment subcategory. These indicators included: 3) Offers enrichment opportunities in a 
variety of content areas; and 4) When provided, athletic programs include both competitive 
and noncompetitive team sports. Although the indicator that focused on a variety of 
enrichment opportunities had a strong mean of 0.87, the one focusing on sports had a low 
mean of 0.40. 

As can be seen in Table 15, the sports indicator was measured using two site staff items 
and two matching observation protocol items. As mentioned previously, the site staff items 
were weighted more heavily than the observation items (2:1), making it necessary for both 
site staff items, or one site staff item and both observation items to meet the two-thirds 
threshold. 
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Table 15 

Descriptive Statistics for Indicator 4 of the Variety of Activities Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Site Staff 


Competitive sports 


15 


83.78 


12.00 


60.00 


100.00 


Noncompetitive sports 


15 


67.78 


18.23 


42.90 


100.00 


Observation 


Competitive Sports 


15 


25.00 


16.37 


0.00 


50.00 


Noncompetitive Sports 


15 


26.67 


30.57 


0.00 


100.00 



Although the she staff hem of eompethive sports had a high mean (M = 83.78), the 
matching observation item was very low (M= 25.00). Likewise, the noncompetitive sports 
item received a moderate mean score from site staff (M= 67.78) and a very low mean score 
from observers (M= 26.67). In other words, the site staff and observers failed to agree about 
the presence of the different types of sports at the program sites. These findings also 
suggested that many of the program sites did not provide a balance between competitive and 
noncompetitive sports. 

The final subcategory for the variety of activities benchmark focused on socialization. 
The sole indicator for this benchmark was: 5) Provides children with regular opportunities 
for socializing. This indicator was strong with a mean of 1.00, demonstrating that 
socialization was encouraged at all 15 program sites. 

Activities support youth development. The final benchmark in the program 
component of instructional features examines whether “activities provide opportunities for 
development of personal responsibility, self-direction, and leadership.” This benchmark was 
comprised of six total indicators: 1) The program promotes youth development; 2) Enables 
participants to develop life skills, resiliency, and self-esteem via activities; 3) Let participants 
take ownership of program selection and development; 4) Provides participants opportunities 
to express their ideas, concerns, and opinions; 5) Program allows for student choice and self- 
direction; and 6) Program promotes the development of leadership abilities. Only two of the 
indicators concerning youth development met the two-thirds rule. Indicators I and 2 each had 
a mean of 1.00, signifying that all 15 sites promoted youth development and supported the 
development of life skills, resiliency, and self-esteem. 

Indicator 3, concerning student involvement in program selection and development, 
resulted in a low mean (M = 0.47). There were five site staff items and three observation 
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items used to examine this indicator. Table 16 shows that four of the staff items were 
moderate to strong (M = 72.62 or above). The exception was the students setting personal 
goals item, which had a low mean of 56.48. The observation items were all low to moderate, 
ranging from 15.00 to 50.00. Because not all of the site staff items met the threshold, and 
none of the observation items met the threshold, this indicator failed to be met. 

Table 16 



Descriptive Statistics for Indicator 3 of the Activities that Support Youth Development Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Site Staff 


Student feedback about setting 


15 


72.62 


19.99 


33.30 


100.00 


Student feedback about activities 


15 


83.10 


12.57 


57.10 


100.00 


Students setting personal goals 


15 


56.48 


25.76 


0.00 


100.00 


Students suggesting activities 


15 


83.75 


12.62 


66.70 


100.00 


Students helping plan activities 


15 


72.98 


18.41 


42.90 


100.00 


Observation 


Student feedback about setting 


15 


15.00 


18.42 


0.00 


50.00 


Student feedback about activities 


15 


50.00 


31.34 


0.00 


100.00 


Students suggesting activities 


15 


21.67 


20.85 


0.00 


75.00 



The indicator regarding providing students with opportunities to express ideas, 
concerns, and opinions also resulted in a low mean (M= 0.60). As shown in Table 17, one 
site staff item and two observation items were used to examine this indicator. Thus, it was 
necessary for the site staff item and at least one of the observation items to meet the two- 
thirds threshold in order for the indicator to meet the two-thirds rule. The table shows that the 
items concerning students reflecting about the program were moderate to weak (M = 72.39 
and M = 16.67, respectively). The remaining item, measuring the acceptance of alternate 
viewpoints, also resulted in a moderate mean (M= 73.33). When examining the responses by 
sites, it was revealed that only 9 of the 15 program sites met the requirement of having two- 
thirds agreement for both the site staff item and one of the observation items. In other words, 
many of the program sites provided limited opportunities for students to express their 
opinions. 
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Table 17 



Descriptive Statistics for Indicator 4 of the Activities that Support Youth Development Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Site Staff 












Students reflect about program 


15 


72.39 


18.18 


33.30 


100.00 


Observation 












Students reflect about program 


15 


16.67 


18.09 


0.00 


50.00 


Acceptance of alternate viewpoints 


15 


1333 


19.97 


25.00 


100.00 


The indicator concerning student choice and self-direction was 


among the weakest for 


the youth development benchmark M = 


0.40). As shown in Table 


18, this indicator was 


examined using two site staff survey and three observation items. Although the table shows 


high means for both of the staff 


items concerning 


student self-direction 


and choice 


(M= 95.65 and M= 80.06, respectively), the means for the observation items were all low to 


very low (ranging from M= 23.33 to M = 


58.33). Because none of the observation items met 


the threshold, the indicator failed to be met. 








Table 18 












Descriptive Statistics for Indicator 5 of the Activities that Support Youth Development Benchmark 


Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Site Staff 












Promote self-direction 


15 


95.65 


8.01 


75.00 


100.00 


Students select activities 


15 


80.06 


16.56 


42.90 


100.00 


Observation 












Students select activities 


15 


58.33 


27.82 


25.00 


100.00 


Student choice (what, how, whom) 


15 


40.00 


28.03 


0.00 


100.00 


Informed/responsible choices 


15 


23.33 


14.84 


0.00 


50.00 



Promoting leadership abilities (see Table 19) was the least prevalent indieator for the 
youth development benehmark, and for the program component of instructional features. 
This indicator had a very low mean of 0.27, indicating that it met the two-thirds rule at only 
four of the program sites. Although the item was examined using three site staff and five 
observation items, only the site staff items received moderate to very high means (ranging 
from 69.26 to 97.47). All of the observation items received means of 48.33 or less. Because 
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of the weighting system (2:1), it was necessary for a combination of site staff and observation 
items to meet the two-thirds threshold in order for the indicator to be met. In other words, the 
indicator failed to be met because site staff and observers at most of the sites did not agree 
about the inclusion of activities that promote leadership. 

Table 19 



Descriptive Statistics for Indicator 6 of the Activities that Support Youth Development Benchmark 



Item 


N 


Mean 


Std. Dev. 


Min. 


Max. 


Site Staff 


Promote leadership abilities 


15 


97.47 


5.26 


85.70 


100.00 


Leadership role during activities 


15 


74.31 


19.30 


42.90 


100.00 


Peer mentoring 


15 


69.26 


18.11 


50.00 


100.00 


Observation 


Leadership role during activities 


15 


48.33 


24.03 


0.00 


100.00 


Promote leadership abilities 


15 


33.33 


26.16 


0.00 


100.00 


Peer mentoring 


15 


20.00 


27.06 


0.00 


75.00 


Child’s initiative 


15 


20.00 


21.55 


0.00 


50.00 


Encourage leadership roles 


15 


28.33 


24.76 


0.00 


75.00 


Setting up the System 


for Instructional Features. As 


with the 


previous two 



components of afterschool effectiveness, the weighting system was used and means were 
calculated for each benchmark. These means represented the aggregated results across the 
15 afterschool sites in this study. Benchmarks that received a score of 7 or higher were 
established as core components, whereas those with a score below 7 were categorized as 
exemplary components. Table 20 shows the means and categorizations of the instructional 
features benchmarks. 

Table 20 



Instructional Features - Mean Scores, Core and High Quality Benchmarks 



Benchmarks 


Mean score 


Core benchmarks 


High quality 
benchmarks 


12. Quality of implementation 


9.90 


P 




13. Variety of activities 


8.60 


P 




14. Activities support youth development 


5.86 




P 
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As shown in the table, two of the three benchmarks concerning instructional features 
were established as core components. The most prevalent benchmarks (as indicated by the 
mean scores) were quality of implementation (M= 9.90) and variety of activities (M= 8.60). 
The only benchmark that could not be established as core involved the inclusion of activities 
that support youth development (M= 5.86). 

In summary, utilizing the results of the weighting system, most of the benchmarks 
extracted from the synthesis of literature were able to be established as core benchmarks of 
quality afterschool programs. This finding was consistent across the three broad components 
of program organization, program environment, and instructional features. In general, over 
two-thirds of the benchmarks for each component were able to be established as core. 
In addition, this study also identified certain program features and practices that afterschool 
programs could employ to further enhance their program quality. For example, enhancements 
could be sought under the benchmarks of family and community involvement, student 
health and well-being, and the activities that support youth development. Although these 
features were not prevalent at most of the 15 sites, these high quality program features were 
identified by a number of current literature, and were considered important for over all 
youth development. 

The Quality Benchmark Rating System 

With the core benchmarks established, the next step was to lay out the format for 
application. The primary objective of this study was to design a preliminary tool^ (Quality 
Benchmark Rating System) that could facilitate policymakers in making research-based 
decisions so that state funding could be applied toward programs that demonstrate promise 
and success. In addition, policymakers could also use the information gathered from the 
Quality Benchmark Rating System to develop and implement appropriate guidelines for 
afterschool policies. Meanwhile, managers of afterschool programs could use this tool to 
fine-tune their objectives and goals, demonstrate accountability for policymakers, and 
promote positive outcomes for students. 

Designing the Quality Benchmark Rating System 

In order to achieve these goals, a comprehensive synthesis of literature was conducted. 
Under the guidance of the literature reviewed, a theoretical model (Figure 1) was established 
to enhance the flexibility of the rating system, and to avoid certain pitfalls that constantly 

^ Based on the identification of the quality benchmarks, this study provides the initial design, sketch, or frame 
work of a tool that can be useful to policy makers, funding agencies, and after school program managers. 
Because this preliminary tool is still in infancy stage, further testing, validation, and refinements are needed 
before it can be put to use broadly. 
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threaten the applications of afterschool studies. The benchmarks and rating system designed 
by this study intended; 

1. To be applicable to all programs serving students of different races, gender, and 
age groups. 

2. To be applicable to programs with different program goals and approaches. 

3. To be applicable to programs with different desired outcomes, such as academic 
achievement, enrichment, etc. 

4. To be applicable to programs run by different organizations such as school 
districts, and community-based and religious based institutions, etc. 

Under this framework, a flexible system was constructed that included planning 
strategies and implementation steps broad enough to encompass the variability of existing 
afterschool programs yet, specific enough to include key components of quality in 
afterschool programming. 

Constructing the Quality Benchmark Rating System 

The Quality Benchmark Rating System was designed under the framework of the 
theoretical model and the structures of the three major components of afterschool programs: 
Program organization, program environment, and instructional features. Each major program 
component has its own checklist for quality as indicated by the associated benchmarks. 

For example, under the Program Organization component there were seven benchmarks 
(program management, program administration, staff support, staff experience and training, 
family involvement, community partnership, and evaluation). First, each individual 
benchmark was provided with a definition under the title to clearly define what the 
benchmark stood for. Next, the associated indicators, as established by the weighting system 
and the two-thirds rule, were listed to the right of each benchmark. Fastly, each indicator also 
had an associated rating score listed to its right. 

The score for each indicator was established by revisiting the literature synthesis 
(see Appendix A). For example, under the benchmark of “management,” a numeral checklist 
with subtitles under each indicator (staff input in decision-making, student input in decision 
making, clear mission statement, and day school and afterschool collaboration) was created 
for the CRESST research team. The researchers then reviewed all the literature/articles 
concerning management in afterschool programs. Each time an indicator was mentioned in 
the literature, a check was marked under the appropriate indicator. When all the literature 
under the benchmark were re-examined, a numerical score was calculated for each indicator. 
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The percentage of the frequency of each indicator mentioned was then rounded to the first 
decimal to form the preliminary score for that indicator. Weekly team discussions on the 
appropriateness of the assigned weight (points) to the rating score were conducted until 
consensus had been reached on all items. 

Tables 21-23 present the organization of the benchmarks and indicators for the Quality 
Benchmark Rating System under each major program components. 
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Table 21 

Program Organization Checklist 



Benchmarks 



Indicators 



Program Management: 1 . Does the program consider staff input in decision-making? 

Program has a collaborative .. ■ . o 

^ ^ ^ ^ 2. Does the program consider student input in decision-makmg? 

management system to meet o t- o 

specific goals outlined in ^ there a clear mission statement present for the program? 
mission statement. 

4. Is there day school and afterschool collaboration? 



Program Administration: 

Program has effective 
management and plan for 
long-term sustainability and 
growth. 



1 . Have program policies been developed for student participation 
and attendance? 

2. Is the budget maintained and adjusted to meet resource needs? 

3. Is a long-term financial plan in place for sustaining and 
fostering program growth? 



□ 2.0 
□ 1.0 

□ 3.5 

□ 3.5 

□ 3.0 

□ 3.0 

□ 4.0 



Staff Support: 

Program staff are given 
adequate support. 



Staff Experience and 
Training: 

All staff members have 
adequate training and 
experience to ensure high 
quality instruction. 



1 . Is the staff well-paid? 

2. Are staff provided performance feedback? 

3. Does staff receive an orientation before working with youth? 

1 . Is there an adequate staff-student ratio? 

2. Is the staff competent in core academic areas? 

3. Does the staff participate in professional development? 

4. Does the program director participate in professional 
development? 

5. Does the staff reflect the cultural diversity of the community? 



Family Involvement: 

Program has a clear plan for 
family involvement. 



1 . Does the staff regularly communicate with parents/families? 

2. Is there a program plan in place for parent involvement? 

3. Are parents provided with opportunities to provide feedback 
about the program? 



□ 


2.0 


□ 


4.0 


□ 


4.0 


□ 


2.0 


□ 


2.0 


□ 


2.0 


□ 


2.0 


□ 


2.0 



□ 5.0 

□ 3.0 

□ 2.0 



Community Partnerships: 1 . Are there established partnerships with local community Q 4.0 

Program engages in organizations/groups? 

community partnerships. 2. Are the youth encouraged to participate in service projects/ 2.0 

programs? 

3. Is there a plan in place for community involvement (e.g., Q 4.0 

volunteering, guest speakers)? 

1 . Is there a method of evaluation for staff performance? Q 2.0 

2. Is there a method of evaluation for program activities? 2.0 

3. Is there a method of evaluation for student engagement? 2.0 

4. Are students’ academic/social skills improvement evaluated? Q 2.0 

5. Are evaluation findings used for program improvement? 2.0 



Evaluation: 

Program has a system in 
place for evaluation of 
students, staff, parents and 
program activities. 
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Table 22 



Program Environment Checklist 



Benchmarks 


Indicators 






Safe Environment: 


1 . Is the program space safe, clean & secure? 


□ 4.00 


Program space is safe, 
clean, and secure. 


2. Is a system in place to keep unauthorized people from 
taking children from program? 


□ 3.00 




3. Are the youth carefully supervised? 


□ 3.00 


Student Health and 


1. Does the program environment enhance students’ health? 


□ 


3.00 


Well-being: 


Program environment 


2. Are healthy and nutritious snacks provided? 


□ 


3.00 


should enhance students’ 


3. Does the staff work hard to minimize health risks? 


n 


3.00 


health. 




4. Is the equipment safe for activity play? 


□ 


1.00 


Well-equipped/ Suitable 
Physical Space: 


1 . Does the program's indoor and outdoor space meet the 
needs of all program activities? 


□ 


3.33 


Program provides physical 


2. Is the space arranged well for a range of activities? 


□ 


3.33 


Space that is appropriately 


equipped and suitable for 


3. Is the space arranged well for simultaneous activities? 


□ 


3.33 


afterschool. 


Positive Relationships: 


Staff-Child Relationship 






Program develops. 


1 . Is there a small child-staff ratio? 


□ 


1.25 


nurtures, and maintains 


positive relationships. 


2. Does the program have guidelines about staff-student 
expectations? 


□ 


1.25 




3. Does the staff relate to children and youth in positive ways? 


□ 


1.25 




4. Does the staff respond appropriately to the individual needs of 
children and youth? 


□ 


1.00 




5. Does the staff encourage children to become more responsible? 


□ 


1.00 




6. Does the staff interact with children to help them learn? 

Child-Child Relationship 


□ 


1.00 




7. Do children interact with one another in positive ways? 

Staff-Staff Relationship 


□ 


1.00 




8. Does the staff work well together to meet the needs of children? 


□ 


0.75 




9. Does the staff communicate with each other while the program is 
in session? 


□ 


0.75 




10. Does the staff provide role models of positive adult 
relationships? 


□ 


0.75 
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Table 23 

Instructional Features Checklist 



Benchmarks 


Indicators 






Quality of 
Implementation: 


1. Are the activities appropriate (i.e., ages, learning styles, and 
abilities) for the children in the program? 


□ 


2.5 


Program provides a 
variety of age-appropriate 
activities that reflect the 


2. Are the activities in line with the interests of the children in 
the program? 


□ 


2.0 


goals and philosophy of 
the program. 


3. Do the activities reflect the languages and cultures of the families 
served? 


□ 


1.5 




4. Do the activities meet the physical, social and emotional 
needs of the students? 


□ 


1.0 




5. Does the program use a variety of instructional methods 
and strategies that reflect current research and policies on 
teaching and learning? 


□ 


2.0 




6. Are children offered multiple opportunities for developing 
and practicing new skills? 


□ 


1.0 



Variety of Activities: 

Program provides a 
balance between 
academics and 
enrichment. 



Core Academics 

1 . Is high quality academic support offered, such as tutoring and 
homework help? 

2. Is instruction offered in a variety of core academic areas? 

Enrichment 

3 . Are there enrichment opportunities in a variety of areas? 

4. When provided, do athletic programs include both competitive 
and noncompetitive team sports? 

Socialization 

5. Are children provided regular opportunities for socializing? 



Activities Support Youth 
Development: 

Activities provide 
opportunities for 
development of personal 
responsibility, self- 
direction, and leadership. 



1 . Does the program promote youth development? 

2. Does the program enable participants to develop life skills, 
resiliency, and self-esteem via activities? 

3. Does the program let participants take ownership of program 
selection and development? 

4. Are participants Provided opportunities to express their ideas, 
concerns, and opinions? 



5. Does the program allow for student choice and self-direction? 



6. Is the development of leadership abilities promoted? 



□ 


1.0 


□ 


3.0 


□ 


3.0 


□ 


1.0 


□ 


2.0 


□ 


1.5 


□ 


1.5 


□ 


2.0 


□ 


1.0 


□ 


2.0 


□ 


2.0 



As shown in Table 23, under eaeh of the major program eomponents, the benehmarks 
and associate indicators were applied as “core” indexes for afterschool program quality, or 
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the necessary conditions for quality afterschool programs. For ease of use, indicators were 
adapted to the form of questions, and appropriate weight or point was assigned. If the 
indicator could be checked off — meaning that particular index of the program quality was 
prevalent at the site — ^points (as shown in the farthest right-hand column) could be allocated. 
In order for a program to a meet a benchmark, it required a minimum score of 7 out of 10 
points. For example, for the benchmark of “management” on program organization (see 
Table 21), if a site/program was determined to have met indicators 2 through 4, it received a 
score of 8 points (1.0 + 3.5 + 3.5) indicating that it had met the program management 
benchmark. If only the first two indicators were determined as having been met, then the site 
received a score of 3 points (2.0 + 1.0) indicating that it had not met the benchmark. For 
programs that desired further improvement, they could examine the indicators that they did 
not check off and make plans on improving program functions in those particular weak areas. 

In order to reduce subjectivity, it was best to have at least 3 raters completing the same 
instrument. On benchmark ratings that did not reach consensus among the raters, discussions 
on those particular benchmarks and indicators would reveal insights and pinpoint areas of 
strength and weakness for program improvements. 

Appendix F presents a Quality Benchmark Rating System Score Sheet where program 
evaluators and policymakers could keep track of the benchmarks and indicators that were 
established through the use of the rating system. 

Features of the Quality Benchmarks Rating System 

As mentioned previously, this rating system could provide policymakers, program 
evaluators, and program managers with an efficient tool for “quality at a glance.” The 
strength and weakness of the afterschool programs could be viewed comprehensively or 
under each of the major program components (program organization, program environment, 
and instructional features), as well as under each benchmark or indicator. This rating system 
was designed as a “quick and efficient solution” to gauge afterschool program quality, while 
at the same time providing “flexibility” for program varieties. 

It should be mentioned that the features of this tool allow the rating system to be 
applicable to a variety of afterschool settings, regardless of whether the focus is on 
academics, health education, physical education, or any other arena. For example, programs 
that do not emphasize core academics could assess whether they offer a variety of activities 
by increasing the points they allocate for enrichment and socialization (while keeping in 
mind the weighting system and two-thirds rule established in this study). Future studies or 
individual programs could also add supplemental scales to this rating system. For example, a 
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science program might want to add another rating seale on key features of high quality 
science curriculum in order to gauge their progress. A health and nutrition program might 
want to add a different seale coneentrating on effective practices that induce healthy habits. 
This flexible feature, and the dual function of serving both as “evaluation at a glance” for 
stakeholders and self-evaluative improvement tool for program managers, differentiates this 
tool from any other assessment instruments for program quality. 

It is important to note here that although the use of this instrument could be valuable to 
a variety of stakeholders, there are limitations as it eurrently exists. Further research is 
needed to validate this tool with a large sample size. As noted previously, additional 
instruetional features could be added to further expand its usefulness. With this study, the 
preliminary Quality Benehmark Rating System provides promise and initiated steps in taking 
research towards the goal of assessing quality for a wide-range of aftersehool programs. 

Discussion 

With eurrent policies and laws that point to the accountability of Extra Learning 
Opportunities (ELO), this study set out to establish standardized expectations (in the form of 
benchmarks) for afterschool programs to run efficiently and effectively. In addition, this 
study designed a preliminary tool (the Quality Benehmark Rating System) to assist 
aftersehool programs in eonsistently evaluating and improving upon the structure and 
implementation of their programs, and examining what works and what needs to be ehanged 
in terms of organization, environment, and instruetion of students. 

The research questions for this study were; 

• What are the basie core benchmarks for quality aftersehool programs serving low- 
income, elementary students? 

• What are the key indicators that help define the core benchmarks in the three main 
components of aftersehool programs (i.e., program organization, program 
environment, and instruetional features)? 

• How can aftersehool programs use a data-based system to identify their own 
strengths and weaknesses, and strive for eontinuous improvement? 

Under these guidelines, a comprehensive synthesis of literature was conducted and a 
theoretieal model was established. Thirteen benchmarks and 63 indicators were extracted 
from the literature; the prevalence of these benehmarks and indicators were further examined 
in 15 high funetioning program sites. Based on these benehmarks and indicators, a Quality 
Benchmark Rating System was designed to assist aftersehool programs in continuous self- 
improvement. The flexibility of the rating system ineludes planning strategies and 
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implementation steps broad enough to encompass the variability of existing afterschool 
programs, yet specific enough to include key components of quality in afterschool 
programming. This system provides program managers, administrators, and staff the capacity 
to gauge whether their program meets basic quality guidelines. Policymakers can also depend 
on the rating system to provide evidences to support their efforts in acquiring increased 
financial support for afterschool programming. As designed, this system can provide 
evidences to generate financial program support, or be used to demonstrate specific program 
needs, such as increased funding for staff training, program resources and supplies, or 
curriculum development. Furthermore, the rating system could also be further expanded to 
meet the needs of programs with specific focuses. For example, for afterschool programs 
with a science focus, benchmarks and indicators for high quality science instructional 
features could be added as a sub-list to assess quality in these specific science elements. 

With current policies and laws that point to the accountability of ELO, data and 
evidence-based programming is essential in assessing program outcomes and improving 
program quality. When programs are consistently evaluated and improved, they can yield the 
most benefits for their student populations and gamer more support from local, state, and 
federal constituencies. 

Implications 

The study findings showed that all 15 sites under study had a clear mission statement, 
and staff inputs were considered strongly in program decision-making. In addition, and as 
expected, these 1 5 sites also demonstrated strong standings on most of the benchmarks under 
the three major components of program organization, program environment, and instmctional 
features. As important as what these sites affirmed, it is also necessary to draw implications 
based on what the researchers learned. The experiences of visiting these high functioning 
program sites accentuated the need for certain benchmarks to be further examined. 

In the area of program organization, the benchmarks of family involvement, as well as 
community partnerships, both documented lower prevalence at the sites. Parent involvement, 
defined as families being welcomed to visit and parents being able to provide feedback, was 
reported to be moderate and weak, respectively. Program directors also stated that there was 
not a clear plan for parent involvement at the sites. Additionally, although parents’ comments 
were welcomed, parents were not given an instrumental role in making important decisions 
within the programs. Similarly, community involvement was also confirmed to be low, 
especially with regard to having community members facilitated as guest speakers, thus 
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decreasing the chances of building stronger partnerships between the program and the larger, 
surrounding community. 

Similar findings were also concluded in a nationwide study (Herman, Huang, & 
Goldschmidt, 2005), especially on parent involvement. Even though literature has 
consistently revealed the importance of parent involvement in their children’s academic 
outcomes (Henderson & Mapp, 2002), a clear relationship between program outcomes and 
parent involvement in afterschool settings has not been established. In fact, very few 
successful afterschool programs could demonstrate that they had high degrees of parent 
involvement, though nearly all demonstrated that they had a high degree of parent 
satisfaction. Further investigation on which elements of parent involvement were the 
contributing factors to student outcomes, or a clearer definition of what one considers parent 
involvement in afterschool settings, are much needed. 

For program environment, the benchmark of student health and well-being also needs 
to be further examined. Many afterschool studies drew on a school effectiveness model to set 
benchmarks for afterschool programs. Because afterschool programs were faced with 
limitations in terms of space (a sick room) and resources (school nurse, nutrition counselor), 
further examination on what should be considered as appropriate or not appropriate in 
afterschool settings ought to be conducted, and appropriate guidelines should be established. 
In this study, the weakness in this benchmark could be partially accounted for, due to the 
handing out of both healthy and unhealthy snacks during nutrition time. There was also a 
lack of providing appropriate guidelines to staff in minimizing healthy risks. For example, 
only about a third of the sites had staff members actively making sure that students were 
washing their hands and separating students when they were ill. 

Easily, in the component of instruction features, the benchmarks on activities that 
support youth development could be further enhanced. Although general forms of support for 
youth development were strongly prevalent across the sites, other, more specific forms of 
youth development were often lacking. For example, most of the programs failed to 
emphasize student ownership of program selection and development. Furthermore, students 
were often not included in setting personal goals, providing suggestions, providing 
comments, or reflecting on the settings and activities of the program. Although the physical 
settings of afterschool programs presented some hindrances in enhancing these goals; at 
times, staff also demonstrated a lot of willingness and desire to promote youth development. 
However, because it is a recently advancing field, many lacked the knowledge and skills to 
promote such concepts. Because positive youth development is the foundation for building 
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good citizenship, this could be a program element that enables policymakers to direct more 
funding toward staff development. 

Conclusion and Policy Implications 

Current literature recognizes the need to identify good practiees in quality-based 
aftersehool programs (Carnegie Couneil on Adoleseent Development, 1994; Beckett et ah, 
2001; Bodily & Beckett, 2005; C. S. Mott Foundation, 2005). Consequently, numerous 
studies have examined indieators and benehmarks that eould define program quality 
(Carnegie Council on Adolescent Development, 1994; Huang, 2001; Goldsmith, Arbreton, & 
Bradshaw, 2004). However, even though quality-based indicators have been identified, 
aftersehool programs still lack a concrete, easily aceessible tool that they can use for self- 
evaluation. At the same time, in order for polieymakers to set realistic expectations for 
aftersehool programs and use evaluations appropriately, they need the evaluative information 
from tested indieators of quality and meaningful recommendations for programs that need 
improvement. 

This study addressed this research gap. Employing a multi-method approach, including 
synthesis of literature, field studies, and data analysis, this study designed a Quality 
Benchmark Rating System to assist polieymakers in decision-making, and to provide 
aftersehool programs with a tool to enhanee their sustainability. 

With aeeess to the Quality Benchmark Rating System, policymakers ean make 
research-based deeisions so that state funding can be applied toward programs that 
demonstrate promise and success. Furthermore, policymakers ean also use the information to 
develop and implement appropriate guidelines for policy involving aftersehool programs. 
Additionally, this system ean assist aftersehool programs in the self-monitoring of their 
progress and apply their effectiveness on their students. Managers of aftersehool programs 
can use this system to fine-tune their objectives and goals, and demonstrate aeeountability for 
policymakers, as well as promote positive outeomes for students. 

Study Limitations 

In this study, the strategy of synthesizing literature was ehosen because few studies 
with qualifying quantitative data or empirical evidences emerged from the literature search. 
In acknowledging the limitations of this process, it is cautioned here that the synthesis results 
are limited in the ability to draw formal inferenees about the larger population. 

Secondly, the newly designed Quality Benchmark Rating System is eonsidered to be at 
an infancy stage. Although the exploratory nature contributes to broad education and policy 
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implications, much more field-testing on this tool is needed. With further testing, refinement, 
and validation eondueted with larger sample sizes, this system eould prove to be a very 
effieient and effeetive tool for aftersehool praetitioners, polieymakers, and a variety of 
stakeholders. 
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Appendix A: 

Literature Summary 

Figure A1 provides a detailed list of the 54 studies that met the criteria for inclusion in 
the synthesis of literature. Each citation within the figure is marked by an “x” to signify the 
quality indicator and broad category (i.e., program organization, program environment, 
instructional features) it addresses. 

As can be seen in the figure, despite observed differences, there are substantial 
overlapping consistencies in opinions across articles. Each benchmark received support from 
at least one-quarter of the sources. Benchmarks that received the strongest levels of support 
(at least 75% or 40 sources) all focused on issues of program organization: Staffing 
/Professional Development, Programming/Activities, and Management/Administration. 
In contrast, those that received moderate support (less than 50% or 27 sources) were 
distributed across the broad categories. 
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Figure Al. Literature summary of afterschool program quality benchmarks {continues on next page). 
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Figure Al. Literature summary of afterschool program quality benchmarks {continues on next page). 
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Appendix B: 
Observation Protocol 



General Background Information 

1 . Date: 

2. Site: 

3. Location: 

4. Observer(s): 



Activity Observation Information 



5. Time Begin: Time End: 

6. Grade level(s): 

7. Location(s) of activity: 

8. Instructional program used (if applicable): 

9. Participants involved in this observation: 







Start of 
Observation 


Mid 

Observation 


End of 
Observation 


a. 


Total # program staff 








b. 


Total # students 








c. 


Total # volunteers 








d. 


Total # day school staff 








e. 


Total # parents 








f. 


Total # other: 
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Academics and Enrichment 



10. Did you observe any of the following aeademic aetivities? 







Start of 
Observation 


Mid 

Observation 


End of 
Observation 


a. 


Homework /Study Hall 








b. 


Reading 








c. 


Language Arts /Writing 








d. 


Math 








e. 


Science 








f. 


History-Social Science 









1 1 . Did you observe any of the following enrichment activities? 







Start of 
Observation 


Mid 

Observation 


End of 
Observation 


a. 


Visual arts (e.g., drawing, arts and crafts) 








b. 


Performing arts (e.g., music, dance, drama) 








c. 


Technology 








d. 


Competitive sports 








e. 


Noncompetitive sports 








f 


Physical fitness 








g- 


Nutrition 








h. 


Study skills 









12. Were students given any free time to socialize? ^No Yes 
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Instructional Methods and Strategies 



13. Did you observe any of the following instruetional praetiees? 

For items a-d, f and i below, instead of putting a check in the appropriate column, please 
note the content area in which an instructional practice took place (e.g., “language arts” 
instead of 







Start of 
Observation 


Mid- 

Observation 


End of 
Observation 


a. 


Project-based learning 








b. 


Experiential learning 








c. 


Real world contexts 








d. 


Learning games 








e. 


Math projects 








f. 


Technology to teach a content area 








g- 


Hands-on science investigations 








h. 


Scientific inquiry 








i. 


Students working in teams/small groups 









14. Evaluate whether instruction was differentiated based on the following: 



Age of the students. 


No 


Somewhat 


Yes 


Skill levels of the students. 


No 


Somewhat 


Yes 



Please elaborate. 



15. Did staff use a variety of strategies to promote student engagement? 
No Yes 



Please elaborate. 
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Personal Responsibility, Self-Direction, and Leadership 



16. Did you observe students doing any of the following? 







Start of 
Observation 


Mid- 

Observation 


End of 
Observation 


a. 


Providing feedback about the program 
setting 








b. 


Providing feedback about the activities 








c. 


Suggesting activities for the program 








d. 


Playing a leadership role during activities 








e. 


Participating in activities that promote 
leadership abilities 








f. 


Selecting the activities in which they want to 
participate 








g- 


Reflecting about their experience in the 
program 








h. 


Acting as peer mentors 









17. Did staff utilize any of the 8 principles of youth development (focusing on building 
student strengths as a way to reduce weaknesses in the areas of physical/psychological 
safety; consistent structure & supervision; supportive relationships; opportunities to 
belong; positive social norms; support for efficacy/mattering; opportunities for skill 
building; and integration of family, school & community)? 

No Yes 



Please elaborate. 



18. Did you witness any instances of staff highlighting student work (e.g., display work, have 
students present work to class)? 

No Yes 



Please elaborate. 
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Collaboration with Day School 



19. Did you observe any communication or collaboration between day school and after 
school staff? No Yes 

If yes, please elaborate. 
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For items 20-22, & 24, note “NA” in the Comments column if this indicator is not 
observable/applicable. 

Safety, Health and Physical Space 

20. Rate the following indicators based on these two questions: 

❖ Does the program provide a safe and healthy environment for all ASP participants? 

❖ Does the program provide physical space that is appropriately equipped and suitable for 
after school activities? 



Comments 


Safety, Health, Physical 
Space 


Not 

Evident 


Somewhat 

Evident 


Moderately 

Evident 


Consistently 

Evident 




The program is operating in a 
site with sufficient security. 












Program space is arranged 
well for a range of activities. 












There are sufficient materials 
available to support program 
activities. 












The program’s indoor space 
meets the needs of students 
(adequate space for number 
of students and variety of 
activities offered). 












The program’s outdoor space 
meets the safety, wellness and 
security needs of students. 












Staff know where the children 
are and what they are doing 
(i.e., an adult is nearby and 
maintaining visual observation 
of students). 












The program provides an 
environment that protects the 
health and enhances the 
wellness of the students 
{i.e.., physical safety). 












Meals and snacks are 
nutritious and adequate in 
portion to meet students’ 
needs. 












Staff ensure that students take 
steps to minimize health risks 
(e.g. hand washing, separate 
sick children, tissues, etc). 
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Code of Conduct (Behavior & Discipline) 



21. Does the program maintain, establish, and communicate a code of conduct? 



Comments 


Code of Conduct 


Not 

Evident 


Somewhat 

Evident 


Moderately 

Evident 


Consistently 

Evident 




Staff use positive techniques to 
guide the behavior of children 
and youth. 












Staff give attention to children 
when they cooperate, share, care 
for materials or join in activities. 












Staff set appropriate limits for 
children. 












Staff use no harsh discipline 
methods. 












Staff encourage children to 
resolve their own conflicts. 
Staff step in only if needed to 
discuss the issues and work 
out a solution. 












Staff applies rewards and 
consequences for student 
behavior appropriately and 
consistently. 












Staff is sensitive to students’ 
respective cultures and 
languages. 
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Relationships 



22. Does the program develop, nurture, and maintain positive relationships among staff and 
students? 



Comments 


Relationships 


Not 

Evident 


Somewhat 

Evident 


Moderately 

Evident 


Consistently 

Evident 




Staff interactions with students 
are characterized by warmth, 
respect, and appreciation for 
their efforts. 












Staff use positive behavior 
management strategies 
(e.g., attend to children who 
are demonstrating positive 
behaviors, consistently 
reinforce expectations for 
student behavior, assist children 
in resolving disputes 
appropriately). 












Students interact with each 
other positively (show respect, 
cooperate, and when problems 
occur use appropriate problem 
solving strategies). 












Staff are actively engaged with 
students (helping them learn, 
participating in conversations 
and activities). 












Staff interactions with each 
other are characterized by 
warmth, respect, and reflect 
appropriate modeling for 
students. 












Staff encourage students to 
make choices. 
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23. Specific things to look for (check general categories if observed, and underline/circle 
specific descriptions if applicable): 

□ Staff relate to all children in positive ways: 

Respect and listen to what they say. 

Make children feel welcome and comfortable. 

Respond to children with acceptance and appreciation. 

Are engaged with children. 

□ Staff respond appropriately to the individual needs of children: 

Show awareness of the special interests and talents of each child. 

Recognize the range of children’s abilities. 

Relate to a child’s culture and home language. 

Respond to the range of feelings and temperaments. 

□ Staff encourage children to make choices and become more responsible: 

Support a child’s initiative. 

Encourage children to take leadership roles. 

Provide children with chances to choose what they will do, how they will do it, and with 
whom. 

Help children make informed and responsible choices. 

□ Staff interact with children to help them learn: 

Ask questions that encourage children to think for themselves. 

Share skills and resources to help children gain information and solve problems. 

Vary the approaches used to help children learn. 

Help children use language skills through frequent conversations. 

□ Children generally interact with one another in positive ways: 

Appear relaxed and involved with each other. 

Show respect for each other. 

Usually cooperate and work well together. 

Try to discuss their differences and work out solutions. 

□ Staff work well together to meet the needs of children: 

Effectively communicate with each other. 

Are cooperative with each other. 

Are respectful of each other. 

Provide role models of positive adult relationships. 
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Overall Climate 



24. What is the program’s overall elimate? 



Climate Indicators 


Not 

Evident 


Somewhat 

Evident 


Moderately 

Evident 


Consistently 

Evident 


Staff are engaged in conversations with 
students and are interested in their lives. 










Staff are relaxed and calm. 










Staff enjoy their work. 










Staff greet students each day. 










Students enjoy participating in the program. 










Activities and staff facilitate students’ 
self confidence. 










Acceptance of alternative viewpoints 
is encouraged. 










Staff build respect for differences 
among students. 











Comments: 
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Appendix C: 
Site Staff Survey 



Thank you for participating in this survey. Your input is especially important for us to 
understand your perceptions about your after school program's environment, 
instructional features and organization. Please be aware that your answers will be kept 
confidential and will not be associated with either your name or after school site in our 
report. 



I. Background Questions 

1 . How many years have you worked as. . . 

a) A staff member in this after school program 

b) A Site Coordinator in this after school program 

2. How many years have you worked at this particular site? 



II. Program Environment 
Healthy and Safe Environment 

Think about the environment in which you work. Please check all that apply: 

□ Students are always under adult supervision. 

□ The program adequately shares approved safety plans and procedures with staff 
and families. 

□ The program regularly performs safety drills. 

□ The program is operating in a site with sufficient security. 

□ The program has effective procedures for arrival and dismissal. 

□ Equipment is safe for activity play. 

□ An effective system is in place to keep imauthorized people from taking students 
from the program. 
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Think about the environment in which you work. Please check all that apply: 

□ Students' health records are adequately maintained in the program. 

□ Students' allergy and medication needs are adequately provided to the staff in 
charge. 

□ The program regularly provides nufrif ious snacks. 

□ The program has adequafe supplies and facilif ies for hand washing. 

□ There are plenfy of opporfunities for sfudenfs fo exercise. 

Program Space and Resources 

Think about the after school program's indoor and outdoor space. Please check all that 
apply: 

The program's indoor space . . . 

□ Adequately meets students' needs. 

□ Has enough room for all program activifies. 

□ Is arranged well for a range of acfivities (e.g., sporfs, creafive arfs, enrichmenf 
offerings, eafing, efc.). 

□ Has adequafe storage space for equipmenf, maferials, and personal possessions. 

□ Allows sfudents to explore their interests. 

□ Allows children to get materials out and put them away by themselves with 
ease. 

The program's outdoor space . . . 

□ Students regularly have a chance to play outdoors. 

□ Students can use a variety of oufdoor equipment and games for bofh acfive and 
quief play. 
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Program Code of Conduct 

Please check all that apply 

□ Program has established clear participation and attendance expectations for 
students. 

□ Staff use posif ive fechniques fo guide sfudenfs' behavior. 

□ Sfaff sef appropriafe limifs for sfudenfs. 

□ Sfaff use no harsh discipline mefhods. 

□ Sfaff encourage sfudenfs fo resolve fheir ownconflicfs. 

□ Sfaff appropriafely and consisfenfly apply rewards and consequences for sfudenf 
behavior. 

□ Sfaff are sensifive fo sfudenfs' respecfive culfure(s) and language(s). 

□ Sfudenfs undersf and fhe rules for behavior. 



Relationships among Staff and Students 



Please mark the extent to which you agree or disagree with each of the following: 





Sfrongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. Program has clear wrif fen guidelines abouf 
expecfafions for sfaff-sfudenf inferacfions. 


1 


2 


3 


4 


2. Acfivities help build relaf ionships befween 
sfaff and sfudenfs. 


1 


2 


3 


4 


3. Sfaff relafe fo all sfudenfs inposifive ways. 


1 


2 


3 


4 


4. Sfaff respond appropriafely fo fhe 
individual needs of sfudenfs. 


1 


2 


3 


4 


5. Sfaff encourage sfudenfs fo make choices and 
become more responsible. 


1 


2 


3 


4 


6. Sfaff inferacf wifh sfudenfs fo help fhem 
learn. 


1 


2 


3 


4 


7. Sfudenfs generally inferacf wifh each ofher 
in posifive ways. 


1 


2 


3 


4 


8. Sfaff work well togefher fo meef fhe needs 
of sfudenfs. 


1 


2 


3 


4 
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Overall Program Climate 

Please mark the extent to which you agree or disagree with each of the following: 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. Staff are engaged in conversations with 


1 






A 


students and are interested in their lives. 




o 




2. Staff enjoy their work. 


1 


2 


3 


4 


3. Staff make an effort to greet students each 


1 


9 




A 


day. 








4. Students enjoy participating in the program. 


1 


2 


3 


4 


5. The program facilitates students' self- 


1 


9 




A 


confidence. 




o 




6. Acceptance of alternative viewpoints is 
encouraged. 


1 


2 


3 


4 


7. Staff build respect for differences among 
students. 


1 


2 


3 


4 



III. Instructional Features 
Activities and Program Goals 



Please mark the extent to which you agree or disagree with each of the following: 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. Activities are offered that reflect the 
different learning styles of the students 
(e.g., visual, auditory, kinesthetic). 


1 


2 


3 


4 


2. Activities reflect the abilities of the 
students in the program. 


1 


2 


3 


4 


3. Activities reflect the interests of the 
students in the program. 


1 


2 


3 


4 


4. Activities are well suited to the age range of 
students in the program. 


1 


2 


3 


4 


5. Activities reflect the preferred language(s) 
of the students. 


1 


2 


3 


4 


6. Activities reflect the different cultures of 
the families served. 


1 


2 


3 


4 
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Academics and Enrichment 



Please mark the extent to which you agree or disagree with each of the following: 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. Students are given adequate assistance with 
their homework. 


1 


2 


3 


4 


2. Students are given regular opportunities to 
socialize. 


1 


2 


3 


4 


3. Students are given sufficient opportunities 
for cognitive development. 


1 


2 


3 


4 


4. Students are provided opportunities to 
develop critical thinking skills. 


1 


2 


3 


4 


5. The program addresses the physical needs 
of the students. 


1 


2 


3 


4 


6. The program addresses the social needs of 
the students. 


1 


2 


3 


4 


7. The program addresses the emotional needs 
of the students. 


1 


2 


3 


4 
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During the past year, which of the following types of activities were offered by your 
program? Please check all that apply. 

□ Reading 

□ Language arts/writing 

□ Math 

□ Science 

□ History-social science 

□ Visual arts (e.g., drawing, arts and crafts) 

□ Performing arts (e.g., music, dance, drama) 

□ Technology 

□ Competitive sports 

□ Noncompetitive sports 

□ Physical fitness 

□ Nutrition 

□ Study skills 

□ Tutoring 



Instructional Methods and Strategies 



Please mark the extent to which you agree or disagree with each of the following: 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. Activities are designed to take into account 
students' strengths and weaknesses. 


1 


2 


3 


4 


2. Instruction is differentiated to meet the 
needs of all students, including those who 
have special needs. 


1 


2 


3 


4 


3. Activities are designed to meet specific 
developmental youth outcomes. 


1 


2 


3 


4 


4. Staff take into account the language and 
culture of the students when designing 
activities. 


1 


2 


3 


4 


5. Students are given multiple opportunities 
to develop new skills. 


1 


2 


3 


4 
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which of the following instructional practices did you use during the past school year? 
Please check all that apply. 

□ Project-based learning practices (students work on projects spanning several days). 

□ Experiential learning (opportunities for hands-on experience, reflection, and 
action). 

□ Utilizing real world contexts. 

□ Teaching math using learning games. 

□ Providing students with opportunities to complete math projects. 

□ Using technology or computers as a tool to teach math. 

□ Having students complete hands-on science investigations. 

□ Having students practice scientific inquiry. 

□ Having students work in small groups. 



Personal Responsibility, Self-Direction, and Leadership 



Please mark the extent to which you agree or disagree with the following: 







Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. 


Staff highlights student work 
(e.g., displays work, has students 
present work to class). 


1 


2 


3 


4 


2. 


Activities help students develop skills that 
will help them succeed in life. 


1 


2 


3 


4 


3. 


Activities promote the development of 
resiliency in students. 


1 


2 


3 


4 


4. 


Activities promote self-esteem. 


1 


2 


3 


4 


5. 


Activities promote self-direction for the 
students. 


1 


2 


3 


4 


6. 


Activities promote leadership abilities. 


1 


2 


3 


4 
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In which of the following ways have students participated in the after school program 
during the past school year? Please check all that apply. 

□ Providing feedback about the program setting. 

□ Providing feedback about the activities offered in the program. 

□ Setting personal goals concerning participation in the program. 

□ Suggesting activities for the program. 

□ Helping staff to plan activities. 

□ Playing a leadership role during activities. 

□ Selecting the activities in which they want to participate. 

□ Reflecting about their experience in the program. 

□ Acting as peer mentors. 



Collaboration with Day School 



Please mark the extent to which you agree or disagree with each of fhe following: 







Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. 


I regularly collaborate with day school staff. 


1 


2 


3 


4 


2. 


A flow of information is maintained between 
the day school and after school program. 


1 


2 


3 


4 


3. 


The day school and after school program 
share staff development opportunities. 


1 


2 


3 


4 


4. 


Day school and after school staff are given 
opportunities to establish relationships. 


1 


2 


3 


4 
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In which ways have you collaborated with day school staff during the past school year? 
Please check all that apply. 

□ Sharing information about curriculum. 

□ Discussing student homework assignments. 

□ Discussing student assessment results. 

□ Sharing instructional strategies. 

□ Identifying complementary learning goals. 

□ Setting goals for individual students. 

□ Discussing successes and challenges. 

□ Discussing the academic progress of my students. 

□ Discussing student behavior. 

□ Monitoring student development. 

□ Developing interdisciplinary projects that span both the day school and after 
school program. 

□ Developing thematic group projects that span both the day school and after school 
program. 

□ Working to ensure that activities support state and local standards. 

□ Working to ensure that activities support state and local benchmarks. 

□ Participating in the school's curriculum plarming committee. 
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IV. Program Organization 
Staff Support 



Please mark the extent to which you agree or disagree with each of the following: 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. There is a clear salary structure in place 
for program staff. 


1 


2 


3 


4 


2. Staff are paid well for the work they do in 
the program. 


1 


2 


3 


4 


3. Full-time staff receive benefits (e.g., 
health insurance, pad leaves of absence). 


1 


2 


3 


4 


4. A written job description outlining 
responsibilities is reviewed with each 
staff member. 


1 


2 


3 


4 


5. Staff are given opportunities to share 
their concerns about the program. 


1 


2 


3 


4 


6. Staff are given continuous feedback on 
their performance. 


1 


2 


3 


4 


7. Staff are given paid preparation time. 


1 


2 


3 


4 


8. Staff are treated as professionals. 


1 


2 


3 


4 


9. Staff are provided with opportunities for 
advancement. 


1 


2 


3 


4 


10. W ith the current staff-student ratio, staff 
are able to give sufficient attention to all 
students. 


1 


2 


3 


4 


11. There is a plan to provide adequate staff 
coverage incase of emergencies or staff 
absences. 


1 


2 


3 


4 
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Staff Experience & Training 



Which of the following professional development (i.e., instructor training) topics did you 
participate in this year through your after school program? Please check all that apply: 

□ Promoting the safety, health, and nutrition of youth. 

□ Promoting students' self-esteem. 

□ Working with families. 

□ Designing activities that support program goals. 

□ Planning activities that support students' developmental needs. 

□ Working with English language learners (students who do not speak English as 
their first language). 

□ Content-specific training (e.g., math, reading, science, art, etc.). 

□ Assessment. 

□ Classroom management and discipline. 

□ Other: 

□ No professional development was offered. 



Thank you for participating!!! 
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Appendix D: 

Program Director Survey 



Thank you for participating in this survey. Your input is especially important for us to 
understand your perceptions about your after school program's organizational 
structure. Please be aware that your answers will be kept confidential and will not be 
associated with either your name or after school site in our report. 



I. Background Questions 

1 . How many years have you worked as. . . 

c) A staff member in this after school program 

d) Program Director in this after school program 

2. How many years have you worked at this particular site? 



Please mark the extent to which you agree or disagree with each of the following 
statements regarding your after school program. 

II. Program Management (planning & policy-making) 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. We have a clear mission statement that 
explains our program's vision. 


1 


2 


3 


4 


2. Staff input is considered in program 
planning. 


1 


2 


3 


4 


3. Student input is considered in program 
planning. 


1 


2 


3 


4 


4. Hours of operation are based on families' 
needs. 


1 


2 


3 


4 


5. I work with the day school principal to 
promote positive change. 


1 


2 


3 


4 


6. I work with the day school principal to 
promote the use of innovation. 


1 


2 


3 


4 
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III. Administration & Program Sustainability 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. There is a long-term financial plan in 
place for susfaining fhe program and 
mainfaining program quality. 


1 


2 


3 


4 


2. All possible resources (e.g., community 
resources, funding) are used to make the 
program affordable for all families. 


1 


2 


3 


4 


3. The program budget is periodically 
adjusted based on resource needs. 


1 


2 


3 


4 


4. There is a marketing plan in place to 
publicize the program. 


1 


2 


3 


4 


5. Stakeholders are kept informed through 
regular progress reports. 


1 


2 


3 


4 


6. All required staff documents are kept 
updated (e.g., insurance, staff security 
clearance, health certification). 


1 


2 


3 


4 


7. All required student documents are kept on 
file (e.g., enrollment, registration). 


1 


2 


3 


4 


8. There is a system in place for monitoring 
student attendance data. 


1 


2 


3 


4 
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IV. Staff Experience & Training 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. We have enough qualified instructors on 
staff to ensure high quality instruction. 


1 


2 


3 


4 


2. Our staff reflects the cultural diversity of 
our community. 


1 


2 


3 


4 


3. Our staff is highly motivated. 


1 


2 


3 


4 


4. Our staff is well-trained for work with 
school-age kids in the after school 
setting. 


1 


2 


3 


4 


5. Our staff is sensitive to the cultural and 
social influences that impact how 
students learn and relate to others. 


1 


2 


3 


4 


6. Staff is competent in core academic areas, 
as appropriate (e.g., math, reading, 
science, art, etc.) 


1 


2 


3 


4 


7. Program directors participate in 

professional development along with 
staff. 


1 


2 


3 


4 


8. Program directors receive training in 
program management. 


1 


2 


3 


4 


9. Qualified volunteers are recmited to 
work in the after school program. 


1 


2 


3 


4 
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V. Family Involvement 



Strongly 

disagree Disagree 


Agree 


Strongly 

agree 


1. Our program has a clearly defined plan ^ 

for parenf involvemenf. 


2 


3 


4 


2. Family members are welcome fo visit any ^ 

time throughout the day. 


2 


3 


4 


3 . We hold orientation sessions for new ^ 

families. 


2 


3 


4 


4 . Staff encourages families to get involved 

in program events (e.g., volimteering, 1 

field trips, events, celebrations). 


2 


3 


4 


5 . Parent input is considered in decisions ^ 

about after school programming. 


2 


3 


4 


6. Staff regularly communicates with 

parents about how they can help their 1 

children learn. 


2 


3 


4 


7 . Staff members are required to 

communicate regularly with parents 1 

about students' well-being. 


2 


3 


4 


8. Staff keeps families informed about what ^ 

is going on in the program. 


2 


3 


4 


9. Parents are aware of attendance policies. 1 


2 


3 


4 


10. Learning opportimities are provided for 

parents through the after school program 1 

(e.g., ESL, computer, literacy workshops). 


2 


3 


4 
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VI. Community Partnerships 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1. We involve local community partners in 
program plarming. 


1 


2 


3 


4 


2. We have established partnerships with 
local community organizations (e.g., 
libraries, businesses, colleges/ 
universities) to support afterschool 
programming. 


1 


2 


3 


4 


3. Our program encourages youth 
representation in local community 
organizations. 


1 


2 


3 


4 


4. We encourage our students to give back to 
the community through community service 
projects. 


1 


2 


3 


4 


5 vVe regularly bring in community members 
as guest speakers. 


1 


2 


3 


4 


6. We recruit volimteers from the 
community. 


1 


2 


3 


4 
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VII. Evaluation 





Strongly 

disagree 


Disagree 


Agree 


Strongly 

agree 


1 . We have an internal method for 
evaluating program activities. 


1 


2 


3 


4 


2. We have an internal method for 
evaluating staff performance. 


1 


2 


3 


4 


3 . We have an internal method for 
evaluating student engagement. 


1 


2 


3 


4 


4 . Students' academic improvement is 
evaluated. 


1 


2 


3 


4 


5 . Students' social skills development is 
evaluated. 


1 


2 


3 


4 


6. Students are surveyed to measure their 
interest in the program. 


1 


2 


3 


4 


7. Parents are surveyed to measure their 
satisfaction with the program. 


1 


2 


3 


4 


8. Staff is surveyed to identify areas for 
program improvement. 


1 


2 


3 


4 


9. Evaluation findings are used to improve 
the program. 


1 


2 


3 


4 


10. Evaluation findings are available to the 
general public (e.g., parents, community). 


1 


2 


3 


4 



Thank you for participating! 
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Appendix E: 
Program Descriptions 



The LA’s BEST Program 

LA’s BEST was implemented in 1998 under the leadership of then Mayor of Los 
Angeles, Tom Bradley, to address the need for adult supervision of young children during the 
hours of 3 to 6 p.m. LA’s BEST is overseen by a cooperative team representing the Office of 
the Mayor of Los Angeles, Los Angeles Unified School District (LAUSD), a board of 
directors, and an advisory board consisting of leaders from business, labor, government, 
education, and the community. 

The program seeks to provide a safe place for children during afterschool hours with 
caring, responsible adults and engaging activities to enrich students’ academic and 
developmental skills. This overall goal is guided by the following five values of LA’s BEST: 

1 . Nothing we do is as important as the effect it has on a child. 

2. Engaging activities develop values, skills, and relationships. Activities are not seen 
as ends in themselves, but as vehicles for creating values, building skills, and 
solidifying peer and adult relationships. An engaging activity is one that holds 
children's attention, awakens their imagination, and inspires them to want to learn 
more. 

3. All children have equal rights to be accepted, respected, and valued by others. 
Children are viewed as individuals to be developed, not problems to be solved. 

4. Children should be involved in decision-making and program design. If children get 
to choose how, when, in what, and with whom to be engaged, they are far more 
likely to enjoy themselves and behave cooperatively. 

5. When we listen for understanding everyone learns — children and adults alike. We 
are constantly able to learn from our children as well as each other. Everyone is a 
learner (EA’s BEST, 2008). 

Student population. The EA’s BEST afterschool program currently serves 26,000 
students in 180 elementary schools in low-income areas, with some of the lowest educational 
resources in the state. Students participating in EA’s BEST are similar to the overall student 
demographics of the greater EAUSD. The majority of students participating in the program 
are Eatina/o, comprising 80% of the total enrolled students. African American students 
comprise 12% of the enrollment and the remaining 8% of students are Asian, Pacific 
Islander, Native American, or White. 

Program structure, EA’s BEST is open to all students at the selected sites and free of 
cost for families. A school selected to be an EA’s BEST program site must meet several 
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criteria, including low performance on academic standardized tests and be located in a low- 
income and/or high crime neighborhood. To become an LA's BEST site, the school principal 
must submit an official letter of request for the program to be implemented at the school. 
Parents enroll their children in LA’s BEST on a first-come, first-serve basis. Teachers can 
also recommend students to participate in the program if they feel the students’ academic 
and/or behavioral skills will especially improve as a result of their attendance. 

In recent years, the LA’s BEST curriculum has been modified to focus on the 
development of the whole child. Thus, a curriculum is in place that focuses on intellectual, 
physical, and social-emotional development. Cognitive skills are developed through 
encouraging responsibility and positive work habits, a love of learning, self-efficacy, and 
fostering future aspirations. Physical and social-emotional development are encouraged 
through a sense of safety and security, a healthy lifestyle, social competence, sense of 
community and a respect for diversity. The LA’s BEST curriculum works towards 
developing these skills by building on the student’s daily life experiences. Thus, each LA’s 
BEST site structures their program to serve the needs of their specific student populations. 
Each site has distinct characteristics and program themes such as arts, self-esteem, conflict 
resolution, and technology. Subsequently, relationships with the day school and community 
support also tend to vary with each site. 

The following list provides an overview of the different educational and enrichment 
activities offered in LA’s BEST program sites: 

Cognitive/Academic. This includes homework time, tutoring, academic incentive 
programs, math and science activities, reading and writing activities, computer activities, and 
psychological programs addressing conflict resolution skills. 

Recreational. This includes arts and crafts, cooking, games, holiday activities, and 
sports such as aerobics, karate, and team sports. 

Performing and Visnal Arts. This includes choir and music, dance, drama/theater, 
fiag/drill team, museum visits, art camps, etc. 

Health and Nntrition. This includes study of nutrition, healthy habits, and exercises 
programs such as tennis, skating, and BEST Fit community health fair. 

Community and Culture. This includes community programs, such as adopt-a- 
grandparent, and community days; and cultural programs, such as those dedicated to Black 
history, “Folklorico,” and other cultural holiday celebrations. 
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Parental involvement activities. These fall under four categories: 

• Celebrations, such as Halloween Kidfest, Community Jam, Awards Days 

• Programs for children, including parent volunteers for daily activities, parent 
attendance of field trips 

• Programs for parents, including parent workshops, guest speakers for parental 
education 

• Communications/information, including open house events, assemblies, parent- 
teacher meetings 

The Pasadena LEARNs Program 

The Pasadena LEARNs afterschool program was established in 1999; jointly created by 
Pasadena Unified School District (PUSD), Partnership for School-Age Children (PSAC), and 
the City of Pasadena. LEARNs is an extended day and year-round program that uses 
academic intervention and enrichment activities as tools for improving students’ academic 
performance. The program is maintained through collaborative efforts of local community 
organizations, principals, teachers, school support staff, parents, students, and community 
activists. The LEARNs program is designed to draw upon the community's strengths and is 
structured to ensure community-wide responsibility, local control, and neighborhood 
ownership. The following three goals guide these program efforts: 

1 . Increase the number of students meeting or exceeding academic standards. 

2. Strengthen student enrichment, leadership, and service opportunities to create 

neighborhood resilience. 

3. Improve student health and safety, and reduce drug use and violence (Pasadena 

Unified School District, 2007). 

Student population. There are 23 schools in PUSD that currently offer the LEARNs 
afterschool program. The majority of LEARNs sites are located at elementary schools. 
In addition, four sites serve middle school students and one site serves middle school 
and high school students. An average of 2,000 children attend the LEARNs program daily. 
The majority of students enrolled at PUSD are Latina/o, comprising 55% of the total enrolled 
students. African-American students comprise 23% of the enrollment at the district. White 
students comprise 15%, and the remaining 7% of the students are Asian, Filipino, Pacific 
Islander, or American Indian. 

Program structure. There is an open enrollment policy for participation in the 
LEARNs program that functions on a first-come, first-serve basis. Family income 
verification determines if a student must pay a fee to participate in the program. For families 
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who do not qualify for their children to participate free of charge, there are minimal sliding 
scale fees, varying from $1 to $5 dollars per day. The program structure is based on the 
“holistic” approach to education that promotes learning and exploring in a safe environment 
in order to achieve greater personal and academic success. LEARNs structures their 
curriculum to be student-centered, building on the skills of the individual child. Each site 
uses PUSD teachers, youth leaders, and community members (program partners) to help 
provide classes for the children at their site. Each school site chooses their own programming 
based on their students' needs and interests, however, most of the sites share similar class 
offerings. 

All LEARNs programs require students to attend three hours daily, from school 
dismissal to 6 p.m. Program sites must dedicate at least 3 hours per week to academic 
enrichment activities that compliment their day school curriculum. An additional requirement 
ensures programs provide reading intervention activities at least 2 hours per week. Reading 
intervention focuses on increasing reading fluency, developing creative writing, reading 
comprehension, and teamwork skills. Many sites offer classes in the following: art, 
computers, cooking, dance, drama, gardening, language arts, leadership, math, music, 
science, sports, and structured recreation. Aside from these activities, most programs provide 
supervised homework assistance and small group tutoring. Following the completion of the 
academic year, LEARNs students showcase their accomplishments for parents, families, and 
teachers. There is also a summer program offered at selected Pasadena LEARNs sites that 
operate a 5- to 6-week session during the summer months with a similar curriculum. 

The Lawndale RAP Program 

Lawndale RAP operates a year-round program. This includes A.M. and P.M. 
KinderCare, as well as before school, afterschool, and intersession programs (during the 
winter, spring and summer) for elementary and middle school children. The program is 
maintained by the Lawndale Elementary School District (LESD) and is overseen by the 
Director of Extended Day Programs for the district. In addition, following the completion of 
the study, Lawndale RAP formed an advisory committee including community partners, 
stakeholders, parents, students, district administrators, and staff. The program seeks to 
provide children with a safe place to spend their time before and afterschool hours. 
Specifically, the program outlines six goals the RAP program strives to meet: 

1 . Helping students reach their amazing potential; 

3. Creating a safe place for students to be every day before and afterschool; 

4. Educating students; 
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5. Enriching students by providing opportunities that they would not otherwise have. 

6. Strengthening the eommunity through families and youth (Lawndale Elementary 
Sehool Distriet, 2005). 

Student population. Lawndale RAP is the smallest of the three programs included in 
this study. Currently, it includes all six elementary schools and both middle schools within 
the district. There are 1200 students participating in the aftersehool eomponent of the 
program eaeh day. The majority of students enrolled at LESD are Latina/o, eomprising 71% 
of the total enrolled students. African-American students comprise 13% of the enrollment at 
the distriet and the remaining 16% of students are Asian, White, Filipino, Pacific Islander, or 
American Indian. 

Program structure. There is an open enrollment poliey for participation in the 
Lawndale RAP program. Parents may pay for their children to attend or they may go on a 
waiting list for a funded spot. Children who have funded spots are required to attend daily 
from 3 to 6 p.m. and may be dropped from the program because of unexeused absenees. 
Onee a student loses their funded seat they must rejoin the waiting list or pay. The RAP 
afterschool program curriculum focuses on three main elements including a nutritional snack, 
homework, and project-based activities. Long-term aetivities foeus on developing aeademie, 
social, and emotional skills such as team building. These activities are aligned with the 
student’s day school curriculum. Other classes that are offered at RAP include, but are not 
limited to music, drama, arts and crafts, cultural activities, science, and cooking. 
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Appendix F: 



Quality Benchmark Score Sheet 



CBO/District: 

Afterschool site; 

Program director: 

Assessment period: From To 



Benchmark 


Score 


Core Quality (Y/N) 


High Quality (Y/N) 


Program Organization 








1. Program management 








2. Program administration 








3. Staff support 








4. Staff experience and training 








5. Family involvement 








6. Community partnerships 








7. Evaluation 








Program Environment 








8. Safe environment 








9. Student health and well-being 








10. Well-equipped and suitable physical space 








1 1 . Positive relationships 








Instructional Features 








1 2 . Quality of imp lementation 








13. Variety of activities 








14. Activities support youth development 









This score sheet provides an overview of program quality at a glance. Each of the 
three major program components are listed with the score of each associated benchmark 
listed. The score sheet also provides a quick reference as to whether the benchmarks are 
considered as “core” or ‘exemplary”. In other words, core benchmarks are used to evaluate 
program quality and exemplary benchmarks are included as indicators for exemplary 
program practices. 
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